Jul 01 18:06:09.332962 ip-10-0-132-145 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 18:06:09.701233 ip-10-0-132-145 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:06:09.701233 ip-10-0-132-145 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 18:06:09.701233 ip-10-0-132-145 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:06:09.701233 ip-10-0-132-145 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 18:06:09.701233 ip-10-0-132-145 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:06:09.702428 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.701977    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 18:06:09.706081 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706059    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:09.706081 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706080    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:09.706081 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706084    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706087    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706090    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706096    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706100    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706103    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706107    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706110    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706113    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706116    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706119    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706121    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706126    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706129    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706132    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706135    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706137    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706140    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706143    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:09.706177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706146    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706150    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706153    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706156    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706159    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706161    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706164    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706167    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706169    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706172    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706175    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706177    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706180    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706184    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706188    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706192    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706195    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706197    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706200    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:09.706646 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706203    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706205    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706208    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706211    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706213    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706216    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706218    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706221    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706224    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706226    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706229    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706231    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706234    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706237    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706240    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706242    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706245    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706247    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706250    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706253    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:09.707099 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706256    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706259    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706262    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706264    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706267    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706270    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706273    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706275    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706278    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706280    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706283    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706286    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706289    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706293    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706295    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706298    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706300    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706302    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706305    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706307    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:09.707590 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706310    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706312    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706315    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706317    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706320    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706323    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706326    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706328    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706331    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706333    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706336    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706338    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706342    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706344    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706347    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706350    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706352    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706355    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.706358    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707165    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:09.708132 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707174    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707177    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707180    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707183    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707186    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707189    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707192    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707194    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707197    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707199    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707202    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707204    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707207    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707210    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707212    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707214    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707217    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707223    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707226    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:09.708607 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707228    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707233    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707237    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707240    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707243    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707246    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707249    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707253    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707257    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707260    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707263    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707265    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707268    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707271    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707273    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707276    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707279    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707281    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707284    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:09.709097 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707287    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707289    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707292    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707294    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707297    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707299    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707301    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707303    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707306    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707308    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707312    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707316    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707319    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707321    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707324    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707326    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707329    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707332    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707334    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707337    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:09.709555 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707340    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707342    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707345    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707347    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707350    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707353    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707357    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707360    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707363    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707365    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707368    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707371    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707375    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707378    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707380    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707384    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707387    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707389    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707392    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:09.710057 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707394    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707396    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707399    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707401    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707405    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707408    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707410    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707413    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707415    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707417    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707421    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707423    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707426    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707429    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707432    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707434    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707437    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707440    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707442    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707444    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:09.710520 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707447    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707864    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707871    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707876    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707879    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707882    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707885    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707888    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707891    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707893    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707896    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707898    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707901    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707903    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707906    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707908    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707913    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707915    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707918    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:09.711015 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707920    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707922    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707925    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707928    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707930    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707933    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707936    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707938    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707941    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707943    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707946    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707948    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707951    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707953    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707955    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707960    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707963    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707966    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707968    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:09.711475 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707971    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707973    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707976    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707978    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707981    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707983    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707986    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707988    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707990    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707993    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707996    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.707999    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708001    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708004    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708006    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708009    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708011    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708014    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708017    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708019    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:09.711962 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708021    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708024    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708026    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708029    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708031    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708033    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708036    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708038    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708040    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708043    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708045    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708048    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708050    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708053    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708056    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708058    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708060    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708064    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708066    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708069    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:09.712780 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708071    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708074    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708077    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708080    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708082    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708084    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708087    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708089    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708093    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708097    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708100    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708102    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708105    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708108    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708110    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708113    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708116    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708118    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708120    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:09.713598 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708123    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708125    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708128    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708203    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708219    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708226    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708231    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708236    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708239    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708243    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708247    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708250    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708255    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708258    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708261    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708266    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708269    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708272    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708275    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708277    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708280    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708284    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708287    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708290    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 18:06:09.714401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708293    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708297    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708300    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708303    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708306    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708309    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708312    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708314    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708317    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708320    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708323    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708327    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708330    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708333    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708336    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708338    2577 flags.go:64] FLAG: --enable-server="true"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708346    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708352    2577 flags.go:64] FLAG: --event-burst="100"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708355    2577 flags.go:64] FLAG: --event-qps="50"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708358    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708364    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708367    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708371    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708375    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708378    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 18:06:09.715314 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708381    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708384    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708387    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708390    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708393    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708395    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708398    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708401    2577 flags.go:64] FLAG: --feature-gates=""
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708405    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708408    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708411    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708413    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708416    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708419    2577 flags.go:64] FLAG: --help="false"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708422    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-132-145.ec2.internal"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708425    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708427    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708430    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708433    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708437    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708439    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708442    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708445    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 18:06:09.716287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708448    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708451    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708455    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708457    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708462    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708465    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708468    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708472    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708476    2577 flags.go:64] FLAG: --lock-file=""
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708478    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708481    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708484    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708490    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708493    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708496    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708498    2577 flags.go:64] FLAG: --logging-format="text"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708501    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708505    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708507    2577 flags.go:64] FLAG: --manifest-url=""
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708510    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708517    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708520    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708525    2577 flags.go:64] FLAG: --max-pods="110"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708528    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708531    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 18:06:09.717295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708534    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708537    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708540    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708543    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708545    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708553    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708556    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708559    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708562    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708565    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708571    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708574    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708578    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708581    2577 flags.go:64] FLAG: --port="10250"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708585    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708588    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d5307ae6a641358a"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708591    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708594    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708597    2577 flags.go:64] FLAG: --register-node="true"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708600    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708603    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708606    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708608    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708625    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708630    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 18:06:09.718362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708633    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708636    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708638    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708641    2577 flags.go:64] FLAG: --runonce="false"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708644    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708647    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708650    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708652    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708655    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708658    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708661    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708664    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708667    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708669    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708672    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708675    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708678    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708682    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708684    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708691    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708693    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708697    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708701    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708704    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708706    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 18:06:09.719536 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708710    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708712    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708715    2577 flags.go:64] FLAG: --v="2"
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708720    2577 flags.go:64] FLAG: --version="false"
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708723    2577 flags.go:64] FLAG: --vmodule=""
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708727    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.708730    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708871    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708875    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708878    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708881    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708884    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708886    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708890    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708892    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708895    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708897    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708899    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708902    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708904    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708907    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:09.720281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708909    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708912    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708914    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708917    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708926    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708930    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708932    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708935    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708938    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708940    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708943    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708945    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708948    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708950    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708953    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708955    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708958    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708960    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708962    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708965    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:09.721166 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708967    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708970    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708972    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708974    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708977    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708979    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708982    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708984    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708987    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708989    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708992    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708995    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708997    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.708999    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709002    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709004    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709007    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709011    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709014    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709016    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:09.721834 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709019    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709021    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709024    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709026    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709029    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709031    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709034    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709036    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709038    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709041    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709043    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709046    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709048    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709051    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709053    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709056    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709059    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709061    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709064    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709066    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:09.722516 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709068    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709071    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709074    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709076    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709078    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709081    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709083    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709086    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709089    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709092    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709095    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709098    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709101    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709104    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709106    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709109    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709111    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709115    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709119    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709121    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:09.723084 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709124    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709127    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709131    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709135    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.709138    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.709143    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.717768    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.717794    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.717822    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.717832    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717920    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717930    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717935    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717940    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:09.723702 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717945    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717949    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717954    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717958    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717962    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717967    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717972    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717976    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717980    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717984    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717988    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717993    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.717997    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718002    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718006    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718010    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718015    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718019    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718023    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718027    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:09.724119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718031    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718035    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718040    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718047    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718053    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718061    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718066    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718071    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718075    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718079    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718083    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718088    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718092    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718096    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718100    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718104    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718108    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718112    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718116    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:09.724660 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718120    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718124    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718128    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718133    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718137    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718141    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718145    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718151    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718157    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718161    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718166    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718170    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718174    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718178    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718182    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718186    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718190    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718194    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718199    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:09.725121 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718204    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718209    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718213    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718218    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718222    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718226    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718230    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718234    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718238    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718242    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718246    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718251    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718257    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718263    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718267    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718272    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718276    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718280    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718284    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718288    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:09.725915 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718293    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718297    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718301    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718305    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718309    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718313    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718317    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718322    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718326    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718331    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718335    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718340    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718344    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718351    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718355    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718359    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:09.726904 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718364    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.718380    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718568    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718578    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718583    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718588    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718592    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718597    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718601    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718605    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718630    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718638    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718643    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718647    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718652    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:06:09.727317 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718657    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718662    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718667    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718672    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718676    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718680    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718684    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718689    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718693    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718697    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718701    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718705    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718710    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718715    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718719    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718724    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718730    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718734    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718737    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:06:09.727716 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718742    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718746    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718750    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718754    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718758    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718762    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718767    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718772    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718778    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718783    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718787    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718793    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718798    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718802    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718807    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718811    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718815    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718818    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718823    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:06:09.728192 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718827    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718831    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718835    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718839    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718844    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718850    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718856    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718861    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718866    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718871    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718876    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718881    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718885    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718890    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718894    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718898    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718903    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718907    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718911    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:06:09.728736 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718915    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718919    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718923    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718927    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718932    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718935    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718939    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718944    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718948    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718952    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718957    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718964    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718968    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718971    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718975    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718978    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718982    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718986    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718990    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718994    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:06:09.729280 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.718998    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.719002    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.719006    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.719009    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.719014    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.719018    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.719022    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.719026    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:09.719030    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.719039    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.719231    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.721992    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.723219    2577 server.go:1019] "Starting client certificate rotation"
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.723329    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 18:06:09.729959 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.723366    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 18:06:09.730340 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.726391    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 18:06:09.731959 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.731935    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 18:06:09.740679 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.740661    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 01 18:06:09.742553 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.742533    2577 log.go:25] "Validated CRI v1 image API"
Jul 01 18:06:09.744809 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.744787    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 18:06:09.747183 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.747160    2577 fs.go:135] Filesystem UUIDs: map[541eadcb-c0fa-4331-9470-9f77a43de920:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f3470ada-7da1-4fc4-8d72-fcc92cefb112:/dev/nvme0n1p4]
Jul 01 18:06:09.747249 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.747183    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 18:06:09.751746 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.751599    2577 manager.go:216] Machine: {Timestamp:2026-07-01 18:06:09.75111283 +0000 UTC m=+0.313283456 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101024 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26be8880cdd82b8856e6bd7b9241d7 SystemUUID:ec26be88-80cd-d82b-8856-e6bd7b9241d7 BootID:a5a001cd-3a5e-4536-8b24-2eafb006338e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f6:d1:15:dc:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f6:d1:15:dc:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:45:0d:18:df:94 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 18:06:09.751746 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.751736    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 18:06:09.751882 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.751869    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 18:06:09.752039 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.752020    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 18:06:09.754875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.754838    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 18:06:09.755030 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.754878    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-145.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 18:06:09.755596 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.755585    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 18:06:09.755654 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.755599    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 18:06:09.755654 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.755606    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 18:06:09.756205 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.756192    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 18:06:09.756898 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.756886    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 18:06:09.759580 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.759563    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 18:06:09.759746 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.759736    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 18:06:09.759778 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.759769    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 18:06:09.759805 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.759780    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 18:06:09.759805 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.759798    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 18:06:09.759867 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.759809    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 01 18:06:09.759867 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.759820    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 18:06:09.760358 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.760345    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 18:06:09.760417 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.760364    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 18:06:09.762438 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.762412    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 18:06:09.763959 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.763945    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 18:06:09.764008 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.763976    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 18:06:09.765129 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765116    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 18:06:09.765164 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765141    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 18:06:09.765164 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765147    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 18:06:09.765164 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765153    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 18:06:09.765164 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765161    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 18:06:09.765269 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765167    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 18:06:09.765269 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765175    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 18:06:09.765269 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765180    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 18:06:09.765269 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765188    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 18:06:09.765269 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765195    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 18:06:09.765269 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765203    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 18:06:09.765269 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765209    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 18:06:09.765904 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765893    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 18:06:09.765904 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.765904    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 18:06:09.769184 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.769170    2577 server.go:1268] "Started kubelet"
Jul 01 18:06:09.769862 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.769297    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 18:06:09.769963 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.769829    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 18:06:09.769963 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.769927    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 18:06:09.770829 ip-10-0-132-145 systemd[1]: Started Kubernetes Kubelet.
Jul 01 18:06:09.772684 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.771082    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 18:06:09.772684 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.772105    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-145.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 18:06:09.773172 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.773135    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 18:06:09.773295 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.773243    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 18:06:09.778047 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.778019    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 18:06:09.778744 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.778723    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-88jh8"
Jul 01 18:06:09.779835 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.779815    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 18:06:09.779926 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.779115    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-145.ec2.internal.18be3ca22fc5974c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-145.ec2.internal,UID:ip-10-0-132-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-145.ec2.internal,},FirstTimestamp:2026-07-01 18:06:09.769142092 +0000 UTC m=+0.331312719,LastTimestamp:2026-07-01 18:06:09.769142092 +0000 UTC m=+0.331312719,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-145.ec2.internal,}"
Jul 01 18:06:09.783060 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.782800    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 18:06:09.783060 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.782815    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 18:06:09.783060 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.782894    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:09.783549 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.783504    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 18:06:09.783647 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.783565    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 18:06:09.783647 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.783576    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 18:06:09.783741 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.783711    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 18:06:09.783741 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.783723    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 18:06:09.784174 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.784133    2577 factory.go:55] Registering systemd factory
Jul 01 18:06:09.784174 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.784151    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 01 18:06:09.784370 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.784359    2577 factory.go:153] Registering CRI-O factory
Jul 01 18:06:09.784370 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.784372    2577 factory.go:223] Registration of the crio container factory successfully
Jul 01 18:06:09.784467 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.784437    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 18:06:09.784467 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.784461    2577 factory.go:103] Registering Raw factory
Jul 01 18:06:09.784543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.784475    2577 manager.go:1196] Started watching for new ooms in manager
Jul 01 18:06:09.784900 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.784887    2577 manager.go:349] Starting recovery of all containers
Jul 01 18:06:09.787161 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.787135    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-88jh8"
Jul 01 18:06:09.792080 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.792045    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 18:06:09.792080 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.792049    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 18:06:09.795032 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.795013    2577 manager.go:354] Recovery completed
Jul 01 18:06:09.799290 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.799275    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:09.802871 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.802849    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:09.802974 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.802886    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:09.802974 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.802898    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:09.803465 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.803451    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 18:06:09.803515 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.803466    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 18:06:09.803515 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.803484    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 18:06:09.803589 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.803515    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:09.804254 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.804236    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:09.804337 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.804272    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:09.804337 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.804297    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:09.804718 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.804707    2577 policy_none.go:49] "None policy: Start"
Jul 01 18:06:09.804765 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.804722    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 18:06:09.804765 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.804745    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 18:06:09.805026 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.804957    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-145.ec2.internal.18be3ca231c83905  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-145.ec2.internal,UID:ip-10-0-132-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-145.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-145.ec2.internal,},FirstTimestamp:2026-07-01 18:06:09.802868997 +0000 UTC m=+0.365039687,LastTimestamp:2026-07-01 18:06:09.802868997 +0000 UTC m=+0.365039687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-145.ec2.internal,}"
Jul 01 18:06:09.805669 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.805659    2577 policy_none.go:47] "Start"
Jul 01 18:06:09.842093 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.842069    2577 manager.go:337] "Starting Device Plugin manager"
Jul 01 18:06:09.842216 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.842115    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 18:06:09.842216 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.842128    2577 server.go:87] "Starting device plugin registration server"
Jul 01 18:06:09.842434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.842419    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 18:06:09.842501 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.842434    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 18:06:09.842551 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.842520    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 18:06:09.842665 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.842604    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 18:06:09.842723 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.842666    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 18:06:09.843218 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.843194    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 18:06:09.843302 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.843253    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:09.861719 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.861534    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 18:06:09.864495 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.863174    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 18:06:09.864495 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.863211    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 18:06:09.864495 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.863233    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 18:06:09.864495 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.863267    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 18:06:09.865198 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.865179    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:09.943596 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.943557    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:09.944605 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.944590    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:09.944673 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.944646    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:09.944673 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.944656    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:09.944731 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.944684    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-145.ec2.internal"
Jul 01 18:06:09.950114 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.950094    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-145.ec2.internal"
Jul 01 18:06:09.950171 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.950122    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-145.ec2.internal\": node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:09.961846 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.961778    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:09.964195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.964174    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal"]
Jul 01 18:06:09.964250 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.964241    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:09.965181 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.965165    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:09.965229 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.965195    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:09.965229 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.965206    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:09.966340 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.966328    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:09.966486 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.966473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:09.966519 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.966503    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:09.967172 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.967154    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:09.967266 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.967188    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:09.967266 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.967202    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:09.967266 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.967200    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:09.967407 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.967275    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:09.967407 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.967287    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:09.968420 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.968404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:09.968520 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.968431    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:06:09.969278 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.969261    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:06:09.969365 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.969289    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:06:09.969365 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.969299    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:06:09.985266 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.985236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/745eb6e171aca622db7fea2abbb34988-config\") pod \"kube-apiserver-proxy-ip-10-0-132-145.ec2.internal\" (UID: \"745eb6e171aca622db7fea2abbb34988\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:09.985378 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.985269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/163abaf6e95031203847852786d95e75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal\" (UID: \"163abaf6e95031203847852786d95e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:09.985378 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:09.985287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/163abaf6e95031203847852786d95e75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal\" (UID: \"163abaf6e95031203847852786d95e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:09.993684 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.993658    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-145.ec2.internal\" not found" node="ip-10-0-132-145.ec2.internal"
Jul 01 18:06:09.998598 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:09.998579    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-145.ec2.internal\" not found" node="ip-10-0-132-145.ec2.internal"
Jul 01 18:06:10.061918 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:10.061889    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:10.086274 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.086251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/163abaf6e95031203847852786d95e75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal\" (UID: \"163abaf6e95031203847852786d95e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:10.086365 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.086285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/745eb6e171aca622db7fea2abbb34988-config\") pod \"kube-apiserver-proxy-ip-10-0-132-145.ec2.internal\" (UID: \"745eb6e171aca622db7fea2abbb34988\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:10.086365 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.086302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/163abaf6e95031203847852786d95e75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal\" (UID: \"163abaf6e95031203847852786d95e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:10.086441 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.086362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/163abaf6e95031203847852786d95e75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal\" (UID: \"163abaf6e95031203847852786d95e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:10.086441 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.086379    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/745eb6e171aca622db7fea2abbb34988-config\") pod \"kube-apiserver-proxy-ip-10-0-132-145.ec2.internal\" (UID: \"745eb6e171aca622db7fea2abbb34988\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:10.086441 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.086362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/163abaf6e95031203847852786d95e75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal\" (UID: \"163abaf6e95031203847852786d95e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:10.162438 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:10.162405    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:10.263246 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:10.263168    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:10.297760 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.297736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:10.302680 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.302660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:10.364298 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:10.364262    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:10.464848 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:10.464810    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:10.565396 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:10.565312    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:10.666027 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:10.665989    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:10.722607 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.722575    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 18:06:10.723079 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.722756    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 18:06:10.767081 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:10.767050    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:10.794652 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.794600    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 18:01:09 +0000 UTC" deadline="2027-12-27 19:48:09.667500862 +0000 UTC"
Jul 01 18:06:10.794652 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.794645    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13057h41m58.872859913s"
Jul 01 18:06:10.847979 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:10.847948    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163abaf6e95031203847852786d95e75.slice/crio-ddbe93b83ba1b35e4a65b4d553ea83d0ecaba131bbb2ed74945a6b5d4b84807f WatchSource:0}: Error finding container ddbe93b83ba1b35e4a65b4d553ea83d0ecaba131bbb2ed74945a6b5d4b84807f: Status 404 returned error can't find the container with id ddbe93b83ba1b35e4a65b4d553ea83d0ecaba131bbb2ed74945a6b5d4b84807f
Jul 01 18:06:10.848212 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:10.848198    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745eb6e171aca622db7fea2abbb34988.slice/crio-306c7f2b27a6af713626e2716d9d07c875667b4915ebc2eebb9f700d45ff4ebe WatchSource:0}: Error finding container 306c7f2b27a6af713626e2716d9d07c875667b4915ebc2eebb9f700d45ff4ebe: Status 404 returned error can't find the container with id 306c7f2b27a6af713626e2716d9d07c875667b4915ebc2eebb9f700d45ff4ebe
Jul 01 18:06:10.852347 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.852327    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:06:10.866833 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.866777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal" event={"ID":"745eb6e171aca622db7fea2abbb34988","Type":"ContainerStarted","Data":"306c7f2b27a6af713626e2716d9d07c875667b4915ebc2eebb9f700d45ff4ebe"}
Jul 01 18:06:10.867144 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:10.867129    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:10.867754 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:10.867735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal" event={"ID":"163abaf6e95031203847852786d95e75","Type":"ContainerStarted","Data":"ddbe93b83ba1b35e4a65b4d553ea83d0ecaba131bbb2ed74945a6b5d4b84807f"}
Jul 01 18:06:10.967240 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:10.967186    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:11.067840 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:11.067786    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:11.168409 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:11.168326    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:11.219160 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.219133    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:11.220101 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.220083    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:11.269229 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:11.269184    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-145.ec2.internal\" not found"
Jul 01 18:06:11.362359 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.362325    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:11.383634 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.383581    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:11.393197 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.393167    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 18:06:11.394071 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.394051    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal"
Jul 01 18:06:11.405793 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.405756    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 18:06:11.684426 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.684246    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:06:11.761879 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.761837    2577 apiserver.go:52] "Watching apiserver"
Jul 01 18:06:11.768135 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.768108    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 18:06:11.770886 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.770846    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-stnmw","kube-system/global-pull-secret-syncer-dkxdw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg","openshift-image-registry/node-ca-hssbk","openshift-multus/multus-additional-cni-plugins-vnxkj","openshift-network-operator/iptables-alerter-mmgrz","openshift-ovn-kubernetes/ovnkube-node-44jxb","kube-system/konnectivity-agent-phk7j","kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rc4mf","openshift-dns/node-resolver-xlgql","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal","openshift-multus/multus-fw7vg","openshift-multus/network-metrics-daemon-qkfdw"]
Jul 01 18:06:11.772671 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.772645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:11.772794 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:11.772727    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:11.773911 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.773463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:11.774866 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.774840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.775941 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.775916    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 18:06:11.776051 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.775916    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 18:06:11.776722 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.776698    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h458w\""
Jul 01 18:06:11.777072 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.777033    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 18:06:11.777636 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.777594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:11.778903 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.778884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.780458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.779520    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 18:06:11.780458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.779607    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2mjvw\""
Jul 01 18:06:11.780458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.779854    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 18:06:11.780458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.779882    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 18:06:11.780458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.779946    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 18:06:11.780458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.780085    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 18:06:11.780458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.780165    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 18:06:11.780458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.780243    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 18:06:11.780458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.780330    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5xrnm\""
Jul 01 18:06:11.780458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.780372    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 18:06:11.780928 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.780587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:11.780928 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.780922    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 18:06:11.781918 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.781211    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 18:06:11.781918 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.781242    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x4l4k\""
Jul 01 18:06:11.781918 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.781304    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 18:06:11.781918 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.781506    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 18:06:11.781918 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.781552    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 18:06:11.782258 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.782053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.784866 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.784130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:11.784866 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.784714    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:06:11.785316 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.785301    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 18:06:11.785658 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.785640    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 18:06:11.785822 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.785806    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 18:06:11.785879 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.785838    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mcfm7\""
Jul 01 18:06:11.786119 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.786103    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 18:06:11.786405 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.786387    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8b58k\""
Jul 01 18:06:11.786500 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.786487    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 18:06:11.786699 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.786682    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 18:06:11.787129 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.787016    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 18:06:11.787129 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.787043    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8d7nz\""
Jul 01 18:06:11.788072 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.787504    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.789006 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.788214    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 18:06:11.789099 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.788373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:11.789284 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:11.789248    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:11.789509 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.789490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.790785 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.790433    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:06:11.790785 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.790453    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 18:06:11.790785 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.790565    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sqxwr\""
Jul 01 18:06:11.791424 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.791396    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bv9zm\""
Jul 01 18:06:11.792232 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.792213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 18:06:11.793334 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.793316    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:11.793411 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:11.793390    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:11.794750 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.794726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fb2f666-408e-4c90-83fb-7386c9dca775-iptables-alerter-script\") pod \"iptables-alerter-mmgrz\" (UID: \"4fb2f666-408e-4c90-83fb-7386c9dca775\") " pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:11.794855 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.794764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79d3fc42-2365-420d-9acd-27da9298db0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.794855 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.794789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.794855 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.794813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-device-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.795032 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.794856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-kubelet\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795032 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.794922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-run-openvswitch\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795032 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.794963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-log-socket\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795032 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.794988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzcd\" (UniqueName: \"kubernetes.io/projected/4fb2f666-408e-4c90-83fb-7386c9dca775-kube-api-access-wjzcd\") pod \"iptables-alerter-mmgrz\" (UID: \"4fb2f666-408e-4c90-83fb-7386c9dca775\") " pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:11.795032 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-cnibin\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.795260 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-sys-fs\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.795260 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q8zx\" (UniqueName: \"kubernetes.io/projected/b55330af-1519-4981-9e03-00da266ea216-kube-api-access-5q8zx\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.795260 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-run-netns\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795260 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-run-ovn\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795260 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-cni-netd\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795260 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795198    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mgq\" (UniqueName: \"kubernetes.io/projected/b1aae139-cdf7-4c0d-b289-13a85e9000d9-kube-api-access-84mgq\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795260 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fb2f666-408e-4c90-83fb-7386c9dca775-host-slash\") pod \"iptables-alerter-mmgrz\" (UID: \"4fb2f666-408e-4c90-83fb-7386c9dca775\") " pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-os-release\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mqp6\" (UniqueName: \"kubernetes.io/projected/79d3fc42-2365-420d-9acd-27da9298db0c-kube-api-access-2mqp6\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-systemd-units\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-run-systemd\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-node-log\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-cni-bin\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-etc-selinux\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1592b6ff-d1f0-4f69-b575-659063252392-agent-certs\") pod \"konnectivity-agent-phk7j\" (UID: \"1592b6ff-d1f0-4f69-b575-659063252392\") " pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859wz\" (UniqueName: \"kubernetes.io/projected/8f91615a-7c42-46c7-b117-fc873db099a8-kube-api-access-859wz\") pod \"node-resolver-xlgql\" (UID: \"8f91615a-7c42-46c7-b117-fc873db099a8\") " pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:11.795592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-var-lib-openvswitch\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-etc-openvswitch\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1aae139-cdf7-4c0d-b289-13a85e9000d9-ovnkube-script-lib\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fc44217-ec85-4856-bd89-c15861990076-serviceca\") pod \"node-ca-hssbk\" (UID: \"1fc44217-ec85-4856-bd89-c15861990076\") " pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgn2\" (UniqueName: \"kubernetes.io/projected/1fc44217-ec85-4856-bd89-c15861990076-kube-api-access-ktgn2\") pod \"node-ca-hssbk\" (UID: \"1fc44217-ec85-4856-bd89-c15861990076\") " pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f91615a-7c42-46c7-b117-fc873db099a8-tmp-dir\") pod \"node-resolver-xlgql\" (UID: \"8f91615a-7c42-46c7-b117-fc873db099a8\") " pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1aae139-cdf7-4c0d-b289-13a85e9000d9-ovn-node-metrics-cert\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fc44217-ec85-4856-bd89-c15861990076-host\") pod \"node-ca-hssbk\" (UID: \"1fc44217-ec85-4856-bd89-c15861990076\") " pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-system-cni-dir\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79d3fc42-2365-420d-9acd-27da9298db0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-registration-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1592b6ff-d1f0-4f69-b575-659063252392-konnectivity-ca\") pod \"konnectivity-agent-phk7j\" (UID: \"1592b6ff-d1f0-4f69-b575-659063252392\") " pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.795983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-slash\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.796015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1aae139-cdf7-4c0d-b289-13a85e9000d9-env-overrides\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.796039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79d3fc42-2365-420d-9acd-27da9298db0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.796175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.796067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-socket-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.796841 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.796103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vxl\" (UniqueName: \"kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl\") pod \"network-check-target-stnmw\" (UID: \"51bddbdc-6a46-4d8c-a6ea-f92883c6189a\") " pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:11.796841 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.796141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f91615a-7c42-46c7-b117-fc873db099a8-hosts-file\") pod \"node-resolver-xlgql\" (UID: \"8f91615a-7c42-46c7-b117-fc873db099a8\") " pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:11.796841 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.796178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.796841 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.796221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1aae139-cdf7-4c0d-b289-13a85e9000d9-ovnkube-config\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.799445 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.799425    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 18:06:11.825480 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.825452    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5gm4n"
Jul 01 18:06:11.836413 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.836387    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5gm4n"
Jul 01 18:06:11.884383 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.884348    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 18:06:11.896903 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.896862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-kubelet-config\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:11.897084 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.896917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fb2f666-408e-4c90-83fb-7386c9dca775-host-slash\") pod \"iptables-alerter-mmgrz\" (UID: \"4fb2f666-408e-4c90-83fb-7386c9dca775\") " pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:11.897084 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.896950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mqp6\" (UniqueName: \"kubernetes.io/projected/79d3fc42-2365-420d-9acd-27da9298db0c-kube-api-access-2mqp6\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.897084 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.896975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-systemd-units\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897084 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.896999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-node-log\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897084 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-cni-bin\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897084 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897023    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fb2f666-408e-4c90-83fb-7386c9dca775-host-slash\") pod \"iptables-alerter-mmgrz\" (UID: \"4fb2f666-408e-4c90-83fb-7386c9dca775\") " pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:11.897084 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-run-k8s-cni-cncf-io\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897091    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-systemd-units\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1592b6ff-d1f0-4f69-b575-659063252392-agent-certs\") pod \"konnectivity-agent-phk7j\" (UID: \"1592b6ff-d1f0-4f69-b575-659063252392\") " pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fc44217-ec85-4856-bd89-c15861990076-serviceca\") pod \"node-ca-hssbk\" (UID: \"1fc44217-ec85-4856-bd89-c15861990076\") " pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f91615a-7c42-46c7-b117-fc873db099a8-tmp-dir\") pod \"node-resolver-xlgql\" (UID: \"8f91615a-7c42-46c7-b117-fc873db099a8\") " pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897161    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-node-log\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897219    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-cni-bin\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjc7l\" (UniqueName: \"kubernetes.io/projected/51b74b41-8fc3-4992-9a5a-f3f01dfde169-kube-api-access-jjc7l\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-os-release\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-system-cni-dir\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-registration-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70bf894a-4512-411a-9b24-0490ec0ec047-cni-binary-copy\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-run-netns\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-sysctl-conf\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.897395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-tuned\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79d3fc42-2365-420d-9acd-27da9298db0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-socket-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vxl\" (UniqueName: \"kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl\") pod \"network-check-target-stnmw\" (UID: \"51bddbdc-6a46-4d8c-a6ea-f92883c6189a\") " pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1aae139-cdf7-4c0d-b289-13a85e9000d9-ovnkube-config\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-var-lib-cni-bin\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx78w\" (UniqueName: \"kubernetes.io/projected/3e73643a-f248-43d2-9433-f8fc9fa89449-kube-api-access-vx78w\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-dbus\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fb2f666-408e-4c90-83fb-7386c9dca775-iptables-alerter-script\") pod \"iptables-alerter-mmgrz\" (UID: \"4fb2f666-408e-4c90-83fb-7386c9dca775\") " pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79d3fc42-2365-420d-9acd-27da9298db0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-device-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-run-openvswitch\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-log-socket\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mgq\" (UniqueName: \"kubernetes.io/projected/b1aae139-cdf7-4c0d-b289-13a85e9000d9-kube-api-access-84mgq\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzcd\" (UniqueName: \"kubernetes.io/projected/4fb2f666-408e-4c90-83fb-7386c9dca775-kube-api-access-wjzcd\") pod \"iptables-alerter-mmgrz\" (UID: \"4fb2f666-408e-4c90-83fb-7386c9dca775\") " pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:11.897998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-cnibin\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q8zx\" (UniqueName: \"kubernetes.io/projected/b55330af-1519-4981-9e03-00da266ea216-kube-api-access-5q8zx\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-run-netns\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-modprobe-d\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-sysconfig\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897907    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fc44217-ec85-4856-bd89-c15861990076-serviceca\") pod \"node-ca-hssbk\" (UID: \"1fc44217-ec85-4856-bd89-c15861990076\") " pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-os-release\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-run-systemd\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.897990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-multus-socket-dir-parent\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-run-multus-certs\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-sys\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-lib-modules\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-etc-selinux\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898110    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f91615a-7c42-46c7-b117-fc873db099a8-tmp-dir\") pod \"node-resolver-xlgql\" (UID: \"8f91615a-7c42-46c7-b117-fc873db099a8\") " pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:11.898543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859wz\" (UniqueName: \"kubernetes.io/projected/8f91615a-7c42-46c7-b117-fc873db099a8-kube-api-access-859wz\") pod \"node-resolver-xlgql\" (UID: \"8f91615a-7c42-46c7-b117-fc873db099a8\") " pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-var-lib-openvswitch\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-etc-openvswitch\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898165    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-system-cni-dir\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1aae139-cdf7-4c0d-b289-13a85e9000d9-ovnkube-script-lib\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898202    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-registration-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70bf894a-4512-411a-9b24-0490ec0ec047-multus-daemon-config\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9jb5\" (UniqueName: \"kubernetes.io/projected/70bf894a-4512-411a-9b24-0490ec0ec047-kube-api-access-t9jb5\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgn2\" (UniqueName: \"kubernetes.io/projected/1fc44217-ec85-4856-bd89-c15861990076-kube-api-access-ktgn2\") pod \"node-ca-hssbk\" (UID: \"1fc44217-ec85-4856-bd89-c15861990076\") " pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1aae139-cdf7-4c0d-b289-13a85e9000d9-ovn-node-metrics-cert\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-cnibin\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898368    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-hostroot\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-etc-kubernetes\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-run\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e73643a-f248-43d2-9433-f8fc9fa89449-tmp\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fc44217-ec85-4856-bd89-c15861990076-host\") pod \"node-ca-hssbk\" (UID: \"1fc44217-ec85-4856-bd89-c15861990076\") " pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:11.899373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79d3fc42-2365-420d-9acd-27da9298db0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1592b6ff-d1f0-4f69-b575-659063252392-konnectivity-ca\") pod \"konnectivity-agent-phk7j\" (UID: \"1592b6ff-d1f0-4f69-b575-659063252392\") " pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-slash\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898581    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1aae139-cdf7-4c0d-b289-13a85e9000d9-env-overrides\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-system-cni-dir\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-var-lib-kubelet\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-systemd\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f91615a-7c42-46c7-b117-fc873db099a8-hosts-file\") pod \"node-resolver-xlgql\" (UID: \"8f91615a-7c42-46c7-b117-fc873db099a8\") " pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898781    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898836    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f91615a-7c42-46c7-b117-fc873db099a8-hosts-file\") pod \"node-resolver-xlgql\" (UID: \"8f91615a-7c42-46c7-b117-fc873db099a8\") " pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898889    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-run-systemd\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-sysctl-d\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-kubelet\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-cni-netd\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-multus-conf-dir\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.900255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.898991    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-etc-selinux\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-kubernetes\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899043    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-etc-openvswitch\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-var-lib-kubelet\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-multus-cni-dir\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-sys-fs\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899146    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79d3fc42-2365-420d-9acd-27da9298db0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-run-ovn\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899195    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-run-ovn\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899301    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-socket-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-var-lib-cni-multus\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-host\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899479    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-sys-fs\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899588    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1aae139-cdf7-4c0d-b289-13a85e9000d9-ovnkube-script-lib\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.899766    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1aae139-cdf7-4c0d-b289-13a85e9000d9-ovnkube-config\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900093    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1aae139-cdf7-4c0d-b289-13a85e9000d9-env-overrides\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900144    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-var-lib-openvswitch\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-os-release\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900281    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fb2f666-408e-4c90-83fb-7386c9dca775-iptables-alerter-script\") pod \"iptables-alerter-mmgrz\" (UID: \"4fb2f666-408e-4c90-83fb-7386c9dca775\") " pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900350    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-run-netns\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900670    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-run-openvswitch\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900715    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-device-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900783    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-log-socket\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900808    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79d3fc42-2365-420d-9acd-27da9298db0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900887    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fc44217-ec85-4856-bd89-c15861990076-host\") pod \"node-ca-hssbk\" (UID: \"1fc44217-ec85-4856-bd89-c15861990076\") " pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.900958    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79d3fc42-2365-420d-9acd-27da9298db0c-cnibin\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.901005    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-slash\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.901067    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b55330af-1519-4981-9e03-00da266ea216-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.901112    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.901233    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-kubelet\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.901267    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79d3fc42-2365-420d-9acd-27da9298db0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.901274    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1aae139-cdf7-4c0d-b289-13a85e9000d9-host-cni-netd\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.901915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.901602    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1592b6ff-d1f0-4f69-b575-659063252392-konnectivity-ca\") pod \"konnectivity-agent-phk7j\" (UID: \"1592b6ff-d1f0-4f69-b575-659063252392\") " pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:11.904546 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.904524    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1592b6ff-d1f0-4f69-b575-659063252392-agent-certs\") pod \"konnectivity-agent-phk7j\" (UID: \"1592b6ff-d1f0-4f69-b575-659063252392\") " pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:11.904789 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.904766    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1aae139-cdf7-4c0d-b289-13a85e9000d9-ovn-node-metrics-cert\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.908810 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.908785    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mqp6\" (UniqueName: \"kubernetes.io/projected/79d3fc42-2365-420d-9acd-27da9298db0c-kube-api-access-2mqp6\") pod \"multus-additional-cni-plugins-vnxkj\" (UID: \"79d3fc42-2365-420d-9acd-27da9298db0c\") " pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:11.912804 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.912774    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mgq\" (UniqueName: \"kubernetes.io/projected/b1aae139-cdf7-4c0d-b289-13a85e9000d9-kube-api-access-84mgq\") pod \"ovnkube-node-44jxb\" (UID: \"b1aae139-cdf7-4c0d-b289-13a85e9000d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:11.913232 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:11.913212    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:11.913312 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:11.913243    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:11.913312 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:11.913257    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z6vxl for pod openshift-network-diagnostics/network-check-target-stnmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:11.913421 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:11.913337    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl podName:51bddbdc-6a46-4d8c-a6ea-f92883c6189a nodeName:}" failed. No retries permitted until 2026-07-01 18:06:12.413306642 +0000 UTC m=+2.975477271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z6vxl" (UniqueName: "kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl") pod "network-check-target-stnmw" (UID: "51bddbdc-6a46-4d8c-a6ea-f92883c6189a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:11.914511 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.914462    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-859wz\" (UniqueName: \"kubernetes.io/projected/8f91615a-7c42-46c7-b117-fc873db099a8-kube-api-access-859wz\") pod \"node-resolver-xlgql\" (UID: \"8f91615a-7c42-46c7-b117-fc873db099a8\") " pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:11.914959 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.914935    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgn2\" (UniqueName: \"kubernetes.io/projected/1fc44217-ec85-4856-bd89-c15861990076-kube-api-access-ktgn2\") pod \"node-ca-hssbk\" (UID: \"1fc44217-ec85-4856-bd89-c15861990076\") " pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:11.915394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.915367    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q8zx\" (UniqueName: \"kubernetes.io/projected/b55330af-1519-4981-9e03-00da266ea216-kube-api-access-5q8zx\") pod \"aws-ebs-csi-driver-node-sxlsg\" (UID: \"b55330af-1519-4981-9e03-00da266ea216\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:11.915905 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.915857    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzcd\" (UniqueName: \"kubernetes.io/projected/4fb2f666-408e-4c90-83fb-7386c9dca775-kube-api-access-wjzcd\") pod \"iptables-alerter-mmgrz\" (UID: \"4fb2f666-408e-4c90-83fb-7386c9dca775\") " pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:11.999961 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.999924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-system-cni-dir\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000102 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:11.999973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-var-lib-kubelet\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000102 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-systemd\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.000102 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-sysctl-d\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.000102 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-multus-conf-dir\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000102 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-kubernetes\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.000102 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-var-lib-kubelet\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000115    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-multus-conf-dir\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-multus-cni-dir\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-var-lib-cni-multus\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-host\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000205    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-sysctl-d\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-kubelet-config\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000254    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-kubelet-config\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-run-k8s-cni-cncf-io\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000279    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-var-lib-cni-multus\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000293    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-multus-cni-dir\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjc7l\" (UniqueName: \"kubernetes.io/projected/51b74b41-8fc3-4992-9a5a-f3f01dfde169-kube-api-access-jjc7l\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000314    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-host\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-os-release\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000347    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-var-lib-kubelet\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70bf894a-4512-411a-9b24-0490ec0ec047-cni-binary-copy\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000353    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-kubernetes\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000069    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-var-lib-kubelet\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.000434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000071    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-system-cni-dir\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-run-netns\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000377    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-run-k8s-cni-cncf-io\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000419    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-run-netns\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000499    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-systemd\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-sysctl-conf\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000563    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-tuned\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-var-lib-cni-bin\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx78w\" (UniqueName: \"kubernetes.io/projected/3e73643a-f248-43d2-9433-f8fc9fa89449-kube-api-access-vx78w\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-dbus\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-modprobe-d\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-sysconfig\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-multus-socket-dir-parent\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-run-multus-certs\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-sys\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-lib-modules\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.001173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70bf894a-4512-411a-9b24-0490ec0ec047-multus-daemon-config\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jb5\" (UniqueName: \"kubernetes.io/projected/70bf894a-4512-411a-9b24-0490ec0ec047-kube-api-access-t9jb5\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-cnibin\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-hostroot\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-etc-kubernetes\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000907    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70bf894a-4512-411a-9b24-0490ec0ec047-cni-binary-copy\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-run\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000929    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-sysconfig\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e73643a-f248-43d2-9433-f8fc9fa89449-tmp\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000992    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-dbus\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.001095    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.001161    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs podName:51b74b41-8fc3-4992-9a5a-f3f01dfde169 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:12.50114644 +0000 UTC m=+3.063317066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs") pod "network-metrics-daemon-qkfdw" (UID: "51b74b41-8fc3-4992-9a5a-f3f01dfde169") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001251    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70bf894a-4512-411a-9b24-0490ec0ec047-multus-daemon-config\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001279    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-sysctl-conf\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.001318    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.001365    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret podName:8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:12.501347398 +0000 UTC m=+3.063518028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret") pod "global-pull-secret-syncer-dkxdw" (UID: "8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001422    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-multus-socket-dir-parent\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-run-multus-certs\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001488    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-modprobe-d\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.002930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001521    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-sys\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.002930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-hostroot\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001603    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-host-var-lib-cni-bin\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001643    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-lib-modules\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.002930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.000884    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-os-release\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001688    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-etc-kubernetes\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001697    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70bf894a-4512-411a-9b24-0490ec0ec047-cnibin\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.002930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.001740    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e73643a-f248-43d2-9433-f8fc9fa89449-run\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.003994 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.003967    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e73643a-f248-43d2-9433-f8fc9fa89449-tmp\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.005405 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.005380    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e73643a-f248-43d2-9433-f8fc9fa89449-etc-tuned\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.009844 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.009812    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9jb5\" (UniqueName: \"kubernetes.io/projected/70bf894a-4512-411a-9b24-0490ec0ec047-kube-api-access-t9jb5\") pod \"multus-fw7vg\" (UID: \"70bf894a-4512-411a-9b24-0490ec0ec047\") " pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.010050 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.010025    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjc7l\" (UniqueName: \"kubernetes.io/projected/51b74b41-8fc3-4992-9a5a-f3f01dfde169-kube-api-access-jjc7l\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:12.010219 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.010199    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx78w\" (UniqueName: \"kubernetes.io/projected/3e73643a-f248-43d2-9433-f8fc9fa89449-kube-api-access-vx78w\") pod \"tuned-rc4mf\" (UID: \"3e73643a-f248-43d2-9433-f8fc9fa89449\") " pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.089445 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.089408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xlgql"
Jul 01 18:06:12.100707 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.100460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:12.108417 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.108394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hssbk"
Jul 01 18:06:12.115270 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.115248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vnxkj"
Jul 01 18:06:12.123971 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.123946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mmgrz"
Jul 01 18:06:12.131461 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.131437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg"
Jul 01 18:06:12.139288 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.139257    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:12.147966 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.147942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rc4mf"
Jul 01 18:06:12.153575 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.153555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fw7vg"
Jul 01 18:06:12.504898 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.504872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vxl\" (UniqueName: \"kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl\") pod \"network-check-target-stnmw\" (UID: \"51bddbdc-6a46-4d8c-a6ea-f92883c6189a\") " pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:12.505062 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.504913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:12.505062 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.504936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:12.505062 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.505043    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:12.505062 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.505047    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:12.505290 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.505061    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:12.505290 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.505085    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:12.505290 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.505098    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z6vxl for pod openshift-network-diagnostics/network-check-target-stnmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:12.505290 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.505099    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret podName:8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:13.505083644 +0000 UTC m=+4.067254262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret") pod "global-pull-secret-syncer-dkxdw" (UID: "8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:12.505290 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.505154    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs podName:51b74b41-8fc3-4992-9a5a-f3f01dfde169 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:13.50513885 +0000 UTC m=+4.067309467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs") pod "network-metrics-daemon-qkfdw" (UID: "51b74b41-8fc3-4992-9a5a-f3f01dfde169") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:12.505290 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:12.505169    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl podName:51bddbdc-6a46-4d8c-a6ea-f92883c6189a nodeName:}" failed. No retries permitted until 2026-07-01 18:06:13.505159938 +0000 UTC m=+4.067330555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6vxl" (UniqueName: "kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl") pod "network-check-target-stnmw" (UID: "51bddbdc-6a46-4d8c-a6ea-f92883c6189a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:12.839152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.839063    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 18:01:11 +0000 UTC" deadline="2028-03-26 00:54:37.259593282 +0000 UTC"
Jul 01 18:06:12.839152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:12.839097    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15198h48m24.420499605s"
Jul 01 18:06:12.877872 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:12.877841    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55330af_1519_4981_9e03_00da266ea216.slice/crio-88a7d7a06a5ea49502b2ed2c92473f987c37473176f70c52af8da0979004d9ff WatchSource:0}: Error finding container 88a7d7a06a5ea49502b2ed2c92473f987c37473176f70c52af8da0979004d9ff: Status 404 returned error can't find the container with id 88a7d7a06a5ea49502b2ed2c92473f987c37473176f70c52af8da0979004d9ff
Jul 01 18:06:12.881126 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:12.881097    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc44217_ec85_4856_bd89_c15861990076.slice/crio-dc2fba714740615457e22d5c9ba35d5a148d2dcda0016f64db4cac26371cff9f WatchSource:0}: Error finding container dc2fba714740615457e22d5c9ba35d5a148d2dcda0016f64db4cac26371cff9f: Status 404 returned error can't find the container with id dc2fba714740615457e22d5c9ba35d5a148d2dcda0016f64db4cac26371cff9f
Jul 01 18:06:13.511168 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.511081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vxl\" (UniqueName: \"kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl\") pod \"network-check-target-stnmw\" (UID: \"51bddbdc-6a46-4d8c-a6ea-f92883c6189a\") " pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:13.511168 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.511126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:13.511168 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.511146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:13.511419 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.511266    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:13.511419 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.511310    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:13.511419 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.511322    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret podName:8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:15.511303857 +0000 UTC m=+6.073474476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret") pod "global-pull-secret-syncer-dkxdw" (UID: "8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:13.511419 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.511335    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:13.511419 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.511348    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z6vxl for pod openshift-network-diagnostics/network-check-target-stnmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:13.511419 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.511345    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:13.511419 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.511399    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl podName:51bddbdc-6a46-4d8c-a6ea-f92883c6189a nodeName:}" failed. No retries permitted until 2026-07-01 18:06:15.511381928 +0000 UTC m=+6.073552544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6vxl" (UniqueName: "kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl") pod "network-check-target-stnmw" (UID: "51bddbdc-6a46-4d8c-a6ea-f92883c6189a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:13.511780 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.511446    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs podName:51b74b41-8fc3-4992-9a5a-f3f01dfde169 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:15.511428953 +0000 UTC m=+6.073599581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs") pod "network-metrics-daemon-qkfdw" (UID: "51b74b41-8fc3-4992-9a5a-f3f01dfde169") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:13.839770 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.839659    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 18:01:11 +0000 UTC" deadline="2028-04-15 10:52:15.905444838 +0000 UTC"
Jul 01 18:06:13.839770 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.839722    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15688h46m2.065727313s"
Jul 01 18:06:13.865488 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.865461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:13.865666 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.865599    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:13.866039 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.866018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:13.866145 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.866126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:13.866201 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.866190    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:13.866280 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:13.866254    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:13.876337 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.876304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg" event={"ID":"b55330af-1519-4981-9e03-00da266ea216","Type":"ContainerStarted","Data":"88a7d7a06a5ea49502b2ed2c92473f987c37473176f70c52af8da0979004d9ff"}
Jul 01 18:06:13.889569 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.889535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal" event={"ID":"745eb6e171aca622db7fea2abbb34988","Type":"ContainerStarted","Data":"02794d01223b8399f5651fffdee6c3262e5f9579c1b4ad09fd76cc5d84012297"}
Jul 01 18:06:13.903260 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.903223    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fw7vg" event={"ID":"70bf894a-4512-411a-9b24-0490ec0ec047","Type":"ContainerStarted","Data":"a73efc03ed71b551da3ffd36258268ac79c1f7666884087a3e140c275607b7e8"}
Jul 01 18:06:13.907324 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.907266    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rc4mf" event={"ID":"3e73643a-f248-43d2-9433-f8fc9fa89449","Type":"ContainerStarted","Data":"df6082e1fbf6588337c055d5b74b9ce91a956918af7628ea71a1e127d68e45a6"}
Jul 01 18:06:13.913544 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.913504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xlgql" event={"ID":"8f91615a-7c42-46c7-b117-fc873db099a8","Type":"ContainerStarted","Data":"11050a4a4e35085eb96327603c497f198813dfa1359ba9c79430f6e4096d8acf"}
Jul 01 18:06:13.919350 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.919295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-phk7j" event={"ID":"1592b6ff-d1f0-4f69-b575-659063252392","Type":"ContainerStarted","Data":"88fe2eb409605f4c63041f99a52469cba7e5d3e3f65dc982e38725bcc2a2576e"}
Jul 01 18:06:13.925291 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.925253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" event={"ID":"b1aae139-cdf7-4c0d-b289-13a85e9000d9","Type":"ContainerStarted","Data":"b0e200e69846c8a1daf4981d769f55151facf1f66161c61ae0dfb84d28052ea3"}
Jul 01 18:06:13.935505 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.935469    2577 generic.go:356] "Generic (PLEG): container finished" podID="163abaf6e95031203847852786d95e75" containerID="ab27768ce63b27617ed49dd170f5f92eaccf2edb172dbcdbcdd614ef85f9304e" exitCode=0
Jul 01 18:06:13.935674 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.935559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal" event={"ID":"163abaf6e95031203847852786d95e75","Type":"ContainerDied","Data":"ab27768ce63b27617ed49dd170f5f92eaccf2edb172dbcdbcdd614ef85f9304e"}
Jul 01 18:06:13.944453 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.944422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mmgrz" event={"ID":"4fb2f666-408e-4c90-83fb-7386c9dca775","Type":"ContainerStarted","Data":"ead2a9e387defc6e29841c7c1b1b21148fd559120e984cc343a4765667df70fd"}
Jul 01 18:06:13.948272 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.948029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnxkj" event={"ID":"79d3fc42-2365-420d-9acd-27da9298db0c","Type":"ContainerStarted","Data":"903f5767efc36632311d3c114fd9ff1efbc3dbfe27d3ff0db99c8de0711e37ea"}
Jul 01 18:06:13.954080 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.954016    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-145.ec2.internal" podStartSLOduration=2.953995222 podStartE2EDuration="2.953995222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:13.902932187 +0000 UTC m=+4.465102825" watchObservedRunningTime="2026-07-01 18:06:13.953995222 +0000 UTC m=+4.516165859"
Jul 01 18:06:13.955703 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:13.955675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hssbk" event={"ID":"1fc44217-ec85-4856-bd89-c15861990076","Type":"ContainerStarted","Data":"dc2fba714740615457e22d5c9ba35d5a148d2dcda0016f64db4cac26371cff9f"}
Jul 01 18:06:14.963776 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:14.963713    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal" event={"ID":"163abaf6e95031203847852786d95e75","Type":"ContainerStarted","Data":"c0831fd921445884bde079cb2432acd3f7453d0ba30a24c0a983a15768fd13a9"}
Jul 01 18:06:15.533712 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:15.530745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vxl\" (UniqueName: \"kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl\") pod \"network-check-target-stnmw\" (UID: \"51bddbdc-6a46-4d8c-a6ea-f92883c6189a\") " pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:15.533712 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:15.530820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:15.533712 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:15.530857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:15.533712 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.531004    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:15.533712 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.531108    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret podName:8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:19.531089105 +0000 UTC m=+10.093259732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret") pod "global-pull-secret-syncer-dkxdw" (UID: "8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:15.533712 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.531655    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:15.533712 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.531713    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs podName:51b74b41-8fc3-4992-9a5a-f3f01dfde169 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:19.531698434 +0000 UTC m=+10.093869063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs") pod "network-metrics-daemon-qkfdw" (UID: "51b74b41-8fc3-4992-9a5a-f3f01dfde169") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:15.533712 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.531788    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:15.533712 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.531809    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:15.534284 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.534094    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z6vxl for pod openshift-network-diagnostics/network-check-target-stnmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:15.534284 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.534160    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl podName:51bddbdc-6a46-4d8c-a6ea-f92883c6189a nodeName:}" failed. No retries permitted until 2026-07-01 18:06:19.534141228 +0000 UTC m=+10.096311845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6vxl" (UniqueName: "kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl") pod "network-check-target-stnmw" (UID: "51bddbdc-6a46-4d8c-a6ea-f92883c6189a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:15.864420 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:15.863887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:15.864420 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:15.863907    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:15.864420 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.864017    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:15.864420 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:15.864066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:15.864420 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.864174    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:15.864420 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:15.864263    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:17.867901 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:17.867867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:17.868322 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:17.867986    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:17.868411 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:17.868364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:17.868543 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:17.868464    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:17.868543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:17.868526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:17.868693 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:17.868646    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:19.568431 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:19.568386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vxl\" (UniqueName: \"kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl\") pod \"network-check-target-stnmw\" (UID: \"51bddbdc-6a46-4d8c-a6ea-f92883c6189a\") " pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:19.568898 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:19.568449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:19.568898 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:19.568478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:19.568898 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.568590    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:19.568898 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.568658    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret podName:8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:27.568641012 +0000 UTC m=+18.130811630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret") pod "global-pull-secret-syncer-dkxdw" (UID: "8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:19.569040 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.569017    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:19.569040 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.569034    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:19.569096 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.569043    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z6vxl for pod openshift-network-diagnostics/network-check-target-stnmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:19.569096 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.569079    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl podName:51bddbdc-6a46-4d8c-a6ea-f92883c6189a nodeName:}" failed. No retries permitted until 2026-07-01 18:06:27.569065427 +0000 UTC m=+18.131236043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6vxl" (UniqueName: "kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl") pod "network-check-target-stnmw" (UID: "51bddbdc-6a46-4d8c-a6ea-f92883c6189a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:19.569159 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.569123    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:19.569159 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.569144    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs podName:51b74b41-8fc3-4992-9a5a-f3f01dfde169 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:27.569137234 +0000 UTC m=+18.131307848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs") pod "network-metrics-daemon-qkfdw" (UID: "51b74b41-8fc3-4992-9a5a-f3f01dfde169") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:19.868327 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:19.867407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:19.868327 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:19.867432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:19.868327 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:19.867584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:19.868327 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.867741    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:19.868327 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.867941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:19.868327 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:19.867944    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:21.864215 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:21.864180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:21.864687 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:21.864323    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:21.864752 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:21.864702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:21.864819 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:21.864792    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:21.864888 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:21.864842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:21.864934 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:21.864918    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:23.867437 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:23.867409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:23.867928 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:23.867517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:23.867928 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:23.867531    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:23.867928 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:23.867643    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:23.867928 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:23.867677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:23.867928 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:23.867737    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:25.867847 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:25.867815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:25.868237 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:25.867822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:25.868237 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:25.867927    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:25.868237 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:25.867940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:25.868237 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:25.868055    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:25.868237 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:25.868135    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:27.629569 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:27.629525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vxl\" (UniqueName: \"kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl\") pod \"network-check-target-stnmw\" (UID: \"51bddbdc-6a46-4d8c-a6ea-f92883c6189a\") " pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:27.630035 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:27.629582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:27.630035 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:27.629628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:27.630035 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.629729    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:27.630035 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.629728    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:27.630035 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.629764    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:27.630035 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.629777    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z6vxl for pod openshift-network-diagnostics/network-check-target-stnmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:27.630035 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.629790    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret podName:8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:43.629770652 +0000 UTC m=+34.191941285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret") pod "global-pull-secret-syncer-dkxdw" (UID: "8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:27.630035 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.629726    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:27.630035 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.629812    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl podName:51bddbdc-6a46-4d8c-a6ea-f92883c6189a nodeName:}" failed. No retries permitted until 2026-07-01 18:06:43.629802535 +0000 UTC m=+34.191973151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6vxl" (UniqueName: "kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl") pod "network-check-target-stnmw" (UID: "51bddbdc-6a46-4d8c-a6ea-f92883c6189a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:27.630035 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.629864    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs podName:51b74b41-8fc3-4992-9a5a-f3f01dfde169 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:43.62984227 +0000 UTC m=+34.192012900 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs") pod "network-metrics-daemon-qkfdw" (UID: "51b74b41-8fc3-4992-9a5a-f3f01dfde169") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:27.863734 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:27.863699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:27.863918 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:27.863760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:27.863918 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:27.863781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:27.863918 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.863859    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:27.864096 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.863951    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:27.864096 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:27.864047    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:29.866287 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:29.866248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:29.866762 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:29.866304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:29.866762 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:29.866248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:29.866762 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:29.866361    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:29.866762 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:29.866463    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:29.866762 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:29.866572    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:31.864548 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:31.864303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:31.865023 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:31.864679    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:31.865023 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:31.864354    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:31.865023 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:31.864303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:31.865023 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:31.864802    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:31.865023 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:31.864921    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:32.000458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.000423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rc4mf" event={"ID":"3e73643a-f248-43d2-9433-f8fc9fa89449","Type":"ContainerStarted","Data":"574edecfba7f07881bf0921add724719aa2a35dd4ad6d1f0ea3404983bf71a12"}
Jul 01 18:06:32.002103 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.002077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xlgql" event={"ID":"8f91615a-7c42-46c7-b117-fc873db099a8","Type":"ContainerStarted","Data":"866f520857e644ca4a96c389f8ad4004ae307e531e7163cb3e55771d0efde7c0"}
Jul 01 18:06:32.004525 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.004496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-phk7j" event={"ID":"1592b6ff-d1f0-4f69-b575-659063252392","Type":"ContainerStarted","Data":"be0f1d9fab5f755fe77360097bf6bc1abdfb8059fc39b56cc9ea3773a2a81380"}
Jul 01 18:06:32.006759 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.006737    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:06:32.007153 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.007126    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1aae139-cdf7-4c0d-b289-13a85e9000d9" containerID="53db5922df5cbc5922a53eeecc2a760caa247dfc9c7485008b7c7c2b588a426c" exitCode=1
Jul 01 18:06:32.007272 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.007212    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" event={"ID":"b1aae139-cdf7-4c0d-b289-13a85e9000d9","Type":"ContainerDied","Data":"53db5922df5cbc5922a53eeecc2a760caa247dfc9c7485008b7c7c2b588a426c"}
Jul 01 18:06:32.007330 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.007304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" event={"ID":"b1aae139-cdf7-4c0d-b289-13a85e9000d9","Type":"ContainerStarted","Data":"6ded6443b1be6a109da43dbbf2b0b69d8792b4d2f7057c715aef5f1642436d00"}
Jul 01 18:06:32.008955 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.008930    2577 generic.go:356] "Generic (PLEG): container finished" podID="79d3fc42-2365-420d-9acd-27da9298db0c" containerID="cd661ed03fb2ac260b521cbdbe7a689e874db5f43ba3f7f3c94a795a03317b7d" exitCode=0
Jul 01 18:06:32.009056 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.009005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnxkj" event={"ID":"79d3fc42-2365-420d-9acd-27da9298db0c","Type":"ContainerDied","Data":"cd661ed03fb2ac260b521cbdbe7a689e874db5f43ba3f7f3c94a795a03317b7d"}
Jul 01 18:06:32.011261 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.010679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hssbk" event={"ID":"1fc44217-ec85-4856-bd89-c15861990076","Type":"ContainerStarted","Data":"4cb78912a084a7a67bbd80456a6651cc7b285a502ef78fd43a7a1ab7c7b8c3f6"}
Jul 01 18:06:32.012273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.012252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg" event={"ID":"b55330af-1519-4981-9e03-00da266ea216","Type":"ContainerStarted","Data":"f248e44bc1539ac8b22478f26d27e096d381f5108c12920b9823a485ec0ff9c0"}
Jul 01 18:06:32.013715 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.013692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fw7vg" event={"ID":"70bf894a-4512-411a-9b24-0490ec0ec047","Type":"ContainerStarted","Data":"48e95b1fa36efc7877077dcbbf79843bde759fb38ac650d22a8ba355795f7760"}
Jul 01 18:06:32.015033 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.014997    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rc4mf" podStartSLOduration=3.3612902780000002 podStartE2EDuration="22.014987484s" totalImagesPullingTime="18.653697206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:32.014874238 +0000 UTC m=+22.577044873" watchObservedRunningTime="2026-07-01 18:06:32.014987484 +0000 UTC m=+22.577158120"
Jul 01 18:06:32.015185 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.015159    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-145.ec2.internal" podStartSLOduration=21.015151781 podStartE2EDuration="21.015151781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:14.979699677 +0000 UTC m=+5.541870319" watchObservedRunningTime="2026-07-01 18:06:32.015151781 +0000 UTC m=+22.577322417"
Jul 01 18:06:32.031138 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.031097    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fw7vg" podStartSLOduration=3.327069323 podStartE2EDuration="22.031084346s" totalImagesPullingTime="18.704015023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:32.030967659 +0000 UTC m=+22.593138295" watchObservedRunningTime="2026-07-01 18:06:32.031084346 +0000 UTC m=+22.593254981"
Jul 01 18:06:32.058209 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.058166    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xlgql" podStartSLOduration=4.408866935 podStartE2EDuration="23.05813447s" totalImagesPullingTime="18.649267535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:32.057821259 +0000 UTC m=+22.619991898" watchObservedRunningTime="2026-07-01 18:06:32.05813447 +0000 UTC m=+22.620305106"
Jul 01 18:06:32.058513 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.058490    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hssbk" podStartSLOduration=11.698922609 podStartE2EDuration="22.058481444s" totalImagesPullingTime="10.359558835s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:32.044149921 +0000 UTC m=+22.606320554" watchObservedRunningTime="2026-07-01 18:06:32.058481444 +0000 UTC m=+22.620652081"
Jul 01 18:06:32.092409 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:32.092355    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-phk7j" podStartSLOduration=7.553034443 podStartE2EDuration="22.092339842s" totalImagesPullingTime="14.539305399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:32.091348168 +0000 UTC m=+22.653518804" watchObservedRunningTime="2026-07-01 18:06:32.092339842 +0000 UTC m=+22.654510477"
Jul 01 18:06:33.018075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:33.017799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mmgrz" event={"ID":"4fb2f666-408e-4c90-83fb-7386c9dca775","Type":"ContainerStarted","Data":"5d1e4132fa35cc9328fe504622d54e7a849c45f502a35dc85a186a5eb71da1ff"}
Jul 01 18:06:33.021006 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:33.020980    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:06:33.021494 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:33.021465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" event={"ID":"b1aae139-cdf7-4c0d-b289-13a85e9000d9","Type":"ContainerStarted","Data":"5baf1bf84958734e036dee95ef7fa74e668db3b338f8effeb752fbaf0474c188"}
Jul 01 18:06:33.021610 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:33.021545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" event={"ID":"b1aae139-cdf7-4c0d-b289-13a85e9000d9","Type":"ContainerStarted","Data":"4710dadc694206b1025d15c49c4062931a887e306603e9f0a3e05b65bf672c49"}
Jul 01 18:06:33.021610 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:33.021589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" event={"ID":"b1aae139-cdf7-4c0d-b289-13a85e9000d9","Type":"ContainerStarted","Data":"da3fe6517817d0193af597e8df8ce61e8bd0c035a179e1a092d2c631517954a2"}
Jul 01 18:06:33.021746 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:33.021660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" event={"ID":"b1aae139-cdf7-4c0d-b289-13a85e9000d9","Type":"ContainerStarted","Data":"5e8d124c68434ce8fd5aeb99669c157529d095ba880e7aab979c9a13882cc59b"}
Jul 01 18:06:33.090836 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:33.090779    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mmgrz" podStartSLOduration=4.438804854 podStartE2EDuration="23.090760383s" totalImagesPullingTime="18.651955529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:33.037190189 +0000 UTC m=+23.599360828" watchObservedRunningTime="2026-07-01 18:06:33.090760383 +0000 UTC m=+23.652931019"
Jul 01 18:06:33.864365 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:33.864323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:33.864525 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:33.864335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:33.864525 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:33.864429    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:33.864597 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:33.864520    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:33.864597 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:33.864552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:33.864678 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:33.864660    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:34.698965 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:34.698919    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:34.699496 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:34.699474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:35.029099 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:35.029032    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:06:35.029445 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:35.029419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" event={"ID":"b1aae139-cdf7-4c0d-b289-13a85e9000d9","Type":"ContainerStarted","Data":"d1f293bb01296fa33f7744790fec0663b9b0a37061bb6db660525bef65ad1a14"}
Jul 01 18:06:35.029581 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:35.029569    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:35.030095 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:35.030069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-phk7j"
Jul 01 18:06:35.866450 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:35.866421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:35.866925 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:35.866421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:35.866925 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:35.866547    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:35.866925 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:35.866424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:35.866925 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:35.866591    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:35.866925 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:35.866698    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:37.036646 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.036347    2577 generic.go:356] "Generic (PLEG): container finished" podID="79d3fc42-2365-420d-9acd-27da9298db0c" containerID="13423aa6592786d1c7139c359ac0ba1677c7c1de03065e37b79ea8fb0335a3bc" exitCode=0
Jul 01 18:06:37.037235 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.036444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnxkj" event={"ID":"79d3fc42-2365-420d-9acd-27da9298db0c","Type":"ContainerDied","Data":"13423aa6592786d1c7139c359ac0ba1677c7c1de03065e37b79ea8fb0335a3bc"}
Jul 01 18:06:37.041767 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.041750    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:06:37.042160 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.042130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" event={"ID":"b1aae139-cdf7-4c0d-b289-13a85e9000d9","Type":"ContainerStarted","Data":"90c86f7d31ada3588193163a90b521a5befa00330e7097ce1ac5c949cc0dc55f"}
Jul 01 18:06:37.042806 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.042785    2577 scope.go:117] "RemoveContainer" containerID="53db5922df5cbc5922a53eeecc2a760caa247dfc9c7485008b7c7c2b588a426c"
Jul 01 18:06:37.363963 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.363824    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 18:06:37.864513 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.863088    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T18:06:37.363958493Z","UUID":"44195e66-06a3-4464-9eab-f65abe470b3d","Handler":null,"Name":"","Endpoint":""}
Jul 01 18:06:37.864513 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.864071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:37.864513 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:37.864197    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:37.864513 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.864569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:37.865341 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:37.864677    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:37.865341 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.864734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:37.865341 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:37.864807    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:37.867323 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.867304    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 18:06:37.867436 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:37.867349    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 18:06:38.049391 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.049304    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:06:38.049950 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.049729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" event={"ID":"b1aae139-cdf7-4c0d-b289-13a85e9000d9","Type":"ContainerStarted","Data":"fac28c51b23a98e63c2ad9e36767963f77b14f152574387b916c9cbb53d27b12"}
Jul 01 18:06:38.050859 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.050833    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:38.054511 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.051540    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:38.054511 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.051838    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:38.058808 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.058701    2577 generic.go:356] "Generic (PLEG): container finished" podID="79d3fc42-2365-420d-9acd-27da9298db0c" containerID="e58a1bc448829b9002e1d9d2cd22af4baabc6fdb612d6294a509c6a31664392f" exitCode=0
Jul 01 18:06:38.058808 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.058802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnxkj" event={"ID":"79d3fc42-2365-420d-9acd-27da9298db0c","Type":"ContainerDied","Data":"e58a1bc448829b9002e1d9d2cd22af4baabc6fdb612d6294a509c6a31664392f"}
Jul 01 18:06:38.080054 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.079327    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb" podStartSLOduration=9.388413786 podStartE2EDuration="28.079304992s" totalImagesPullingTime="18.690891206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:38.078418759 +0000 UTC m=+28.640589396" watchObservedRunningTime="2026-07-01 18:06:38.079304992 +0000 UTC m=+28.641475629"
Jul 01 18:06:38.081651 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.081371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg" event={"ID":"b55330af-1519-4981-9e03-00da266ea216","Type":"ContainerStarted","Data":"2ce9b15a6c37993150579e2d44d3ada412e388143d0183a571fdf9513583e9b7"}
Jul 01 18:06:38.088607 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.088415    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:38.088607 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.088552    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:06:38.647662 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.647609    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-stnmw"]
Jul 01 18:06:38.647871 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.647757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:38.647871 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:38.647845    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:38.648311 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.648289    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dkxdw"]
Jul 01 18:06:38.648404 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.648395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:38.648508 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:38.648490    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:38.650130 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.650105    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qkfdw"]
Jul 01 18:06:38.650232 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:38.650196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:38.650296 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:38.650277    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:39.085755 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:39.085669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg" event={"ID":"b55330af-1519-4981-9e03-00da266ea216","Type":"ContainerStarted","Data":"d3d994090cd8a58fea571cdfd0e901241c523e75db28b2f2b0e4deb11688f10e"}
Jul 01 18:06:39.105530 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:39.105479    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sxlsg" podStartSLOduration=4.23286828 podStartE2EDuration="29.105464424s" totalImagesPullingTime="24.872596144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:39.105380435 +0000 UTC m=+29.667551070" watchObservedRunningTime="2026-07-01 18:06:39.105464424 +0000 UTC m=+29.667635037"
Jul 01 18:06:40.863764 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:40.863733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:40.864116 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:40.863739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:40.864116 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:40.863865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:40.864116 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:40.863848    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:40.864116 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:40.863947    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:40.864116 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:40.864014    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:42.863701 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:42.863663    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:42.863701 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:42.863688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:42.864395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:42.863666    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:42.864395 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:42.863792    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-stnmw" podUID="51bddbdc-6a46-4d8c-a6ea-f92883c6189a"
Jul 01 18:06:42.864395 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:42.863868    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkfdw" podUID="51b74b41-8fc3-4992-9a5a-f3f01dfde169"
Jul 01 18:06:42.864395 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:42.863917    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dkxdw" podUID="8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787"
Jul 01 18:06:42.887642 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:42.887604    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xlgql_8f91615a-7c42-46c7-b117-fc873db099a8/dns-node-resolver/0.log"
Jul 01 18:06:43.097979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:43.097952    2577 generic.go:356] "Generic (PLEG): container finished" podID="79d3fc42-2365-420d-9acd-27da9298db0c" containerID="88adf9910036ad55d1457c34d38c84ab2d555e0ffb7f059c76ad3b8a315c6e25" exitCode=0
Jul 01 18:06:43.098128 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:43.098008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnxkj" event={"ID":"79d3fc42-2365-420d-9acd-27da9298db0c","Type":"ContainerDied","Data":"88adf9910036ad55d1457c34d38c84ab2d555e0ffb7f059c76ad3b8a315c6e25"}
Jul 01 18:06:43.645478 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:43.645439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vxl\" (UniqueName: \"kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl\") pod \"network-check-target-stnmw\" (UID: \"51bddbdc-6a46-4d8c-a6ea-f92883c6189a\") " pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:43.645478 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:43.645481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:43.645725 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:43.645504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:43.645725 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:43.645597    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:43.645725 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:43.645679    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs podName:51b74b41-8fc3-4992-9a5a-f3f01dfde169 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:15.645664778 +0000 UTC m=+66.207835395 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs") pod "network-metrics-daemon-qkfdw" (UID: "51b74b41-8fc3-4992-9a5a-f3f01dfde169") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:06:43.645725 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:43.645597    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:43.645878 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:43.645752    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret podName:8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:15.645739469 +0000 UTC m=+66.207910084 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret") pod "global-pull-secret-syncer-dkxdw" (UID: "8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:06:43.645878 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:43.645599    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:06:43.645878 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:43.645775    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:06:43.645878 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:43.645784    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z6vxl for pod openshift-network-diagnostics/network-check-target-stnmw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:43.645878 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:43.645810    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl podName:51bddbdc-6a46-4d8c-a6ea-f92883c6189a nodeName:}" failed. No retries permitted until 2026-07-01 18:07:15.645804068 +0000 UTC m=+66.207974682 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z6vxl" (UniqueName: "kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl") pod "network-check-target-stnmw" (UID: "51bddbdc-6a46-4d8c-a6ea-f92883c6189a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:06:43.673223 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:43.673188    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hssbk_1fc44217-ec85-4856-bd89-c15861990076/node-ca/0.log"
Jul 01 18:06:44.193836 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.193811    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-145.ec2.internal" event="NodeReady"
Jul 01 18:06:44.194175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.193953    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 18:06:44.272881 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.272846    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fjkjl"]
Jul 01 18:06:44.295233 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.295197    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7mtzl"]
Jul 01 18:06:44.295405 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.295339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.299217 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.298528    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9gc4n\""
Jul 01 18:06:44.299217 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.298956    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 18:06:44.299427 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.299238    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 18:06:44.299427 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.299308    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 18:06:44.299529 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.299238    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 18:06:44.306843 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.306827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7mtzl"]
Jul 01 18:06:44.306916 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.306853    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fjkjl"]
Jul 01 18:06:44.306957 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.306932    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7mtzl"
Jul 01 18:06:44.309261 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.309239    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 18:06:44.309373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.309288    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 18:06:44.309373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.309320    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fptjw\""
Jul 01 18:06:44.309373 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.309289    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 18:06:44.355517 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.355490    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rx2b6"]
Jul 01 18:06:44.373212 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.373188    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rx2b6"]
Jul 01 18:06:44.373344 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.373305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.376292 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.376268    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r7nz8\""
Jul 01 18:06:44.376425 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.376369    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 18:06:44.376425 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.376399    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 18:06:44.451561 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4fk\" (UniqueName: \"kubernetes.io/projected/89bd1a5c-5b4a-4212-b175-2157b3550774-kube-api-access-jz4fk\") pod \"ingress-canary-7mtzl\" (UID: \"89bd1a5c-5b4a-4212-b175-2157b3550774\") " pod="openshift-ingress-canary/ingress-canary-7mtzl"
Jul 01 18:06:44.451561 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafab7ef-f502-4354-a7aa-d42d504684dd-tmp-dir\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.451561 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.451786 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89bd1a5c-5b4a-4212-b175-2157b3550774-cert\") pod \"ingress-canary-7mtzl\" (UID: \"89bd1a5c-5b4a-4212-b175-2157b3550774\") " pod="openshift-ingress-canary/ingress-canary-7mtzl"
Jul 01 18:06:44.451786 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-data-volume\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.451786 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7499\" (UniqueName: \"kubernetes.io/projected/aafab7ef-f502-4354-a7aa-d42d504684dd-kube-api-access-n7499\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.451786 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aafab7ef-f502-4354-a7aa-d42d504684dd-config-volume\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.451786 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aafab7ef-f502-4354-a7aa-d42d504684dd-metrics-tls\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.451786 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-crio-socket\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.451953 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.451953 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.451837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8wvp\" (UniqueName: \"kubernetes.io/projected/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-api-access-z8wvp\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.552858 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.552826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.552858 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.552863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8wvp\" (UniqueName: \"kubernetes.io/projected/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-api-access-z8wvp\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.553059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.552886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4fk\" (UniqueName: \"kubernetes.io/projected/89bd1a5c-5b4a-4212-b175-2157b3550774-kube-api-access-jz4fk\") pod \"ingress-canary-7mtzl\" (UID: \"89bd1a5c-5b4a-4212-b175-2157b3550774\") " pod="openshift-ingress-canary/ingress-canary-7mtzl"
Jul 01 18:06:44.553059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.552903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafab7ef-f502-4354-a7aa-d42d504684dd-tmp-dir\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.553059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.552930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.553059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.552950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89bd1a5c-5b4a-4212-b175-2157b3550774-cert\") pod \"ingress-canary-7mtzl\" (UID: \"89bd1a5c-5b4a-4212-b175-2157b3550774\") " pod="openshift-ingress-canary/ingress-canary-7mtzl"
Jul 01 18:06:44.553059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.552967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-data-volume\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.553059 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:44.552986    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:06:44.553059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.552995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7499\" (UniqueName: \"kubernetes.io/projected/aafab7ef-f502-4354-a7aa-d42d504684dd-kube-api-access-n7499\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.553059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.553021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aafab7ef-f502-4354-a7aa-d42d504684dd-config-volume\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.553059 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:44.553040    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:06:44.553059 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:44.553054    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls podName:98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:45.053034719 +0000 UTC m=+35.615205352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjkjl" (UID: "98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:06:44.553539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.553077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aafab7ef-f502-4354-a7aa-d42d504684dd-metrics-tls\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.553539 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:44.553102    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm podName:98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:45.053085474 +0000 UTC m=+35.615256092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fjkjl" (UID: "98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34") : configmap "kube-rbac-proxy" not found
Jul 01 18:06:44.553539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.553149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-crio-socket\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.553539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.553388    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-crio-socket\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.553539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.553413    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-data-volume\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.553539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.553413    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafab7ef-f502-4354-a7aa-d42d504684dd-tmp-dir\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.553802 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.553706    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aafab7ef-f502-4354-a7aa-d42d504684dd-config-volume\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.557054 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.557032    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aafab7ef-f502-4354-a7aa-d42d504684dd-metrics-tls\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.557170 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.557154    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89bd1a5c-5b4a-4212-b175-2157b3550774-cert\") pod \"ingress-canary-7mtzl\" (UID: \"89bd1a5c-5b4a-4212-b175-2157b3550774\") " pod="openshift-ingress-canary/ingress-canary-7mtzl"
Jul 01 18:06:44.561658 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.561600    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7499\" (UniqueName: \"kubernetes.io/projected/aafab7ef-f502-4354-a7aa-d42d504684dd-kube-api-access-n7499\") pod \"dns-default-rx2b6\" (UID: \"aafab7ef-f502-4354-a7aa-d42d504684dd\") " pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.561782 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.561695    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8wvp\" (UniqueName: \"kubernetes.io/projected/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-api-access-z8wvp\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:44.561870 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.561851    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4fk\" (UniqueName: \"kubernetes.io/projected/89bd1a5c-5b4a-4212-b175-2157b3550774-kube-api-access-jz4fk\") pod \"ingress-canary-7mtzl\" (UID: \"89bd1a5c-5b4a-4212-b175-2157b3550774\") " pod="openshift-ingress-canary/ingress-canary-7mtzl"
Jul 01 18:06:44.615456 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.615419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7mtzl"
Jul 01 18:06:44.682497 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.682433    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:44.816904 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.816721    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7mtzl"]
Jul 01 18:06:44.821008 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:44.820979    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bd1a5c_5b4a_4212_b175_2157b3550774.slice/crio-84ee81e1794567a5b4d9a77083a0c1ef1ac50e4bac83cd1ec04718d62718f173 WatchSource:0}: Error finding container 84ee81e1794567a5b4d9a77083a0c1ef1ac50e4bac83cd1ec04718d62718f173: Status 404 returned error can't find the container with id 84ee81e1794567a5b4d9a77083a0c1ef1ac50e4bac83cd1ec04718d62718f173
Jul 01 18:06:44.846225 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.846196    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rx2b6"]
Jul 01 18:06:44.849750 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:44.849726    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafab7ef_f502_4354_a7aa_d42d504684dd.slice/crio-ac7c5c41ad27728a6426ec3d42dff1de32e589f49974f6e363d534479d51f887 WatchSource:0}: Error finding container ac7c5c41ad27728a6426ec3d42dff1de32e589f49974f6e363d534479d51f887: Status 404 returned error can't find the container with id ac7c5c41ad27728a6426ec3d42dff1de32e589f49974f6e363d534479d51f887
Jul 01 18:06:44.863874 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.863849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:06:44.863985 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.863883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:06:44.864037 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.864026    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:06:44.866473 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.866450    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 18:06:44.866590 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.866537    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 18:06:44.866590 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.866542    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 18:06:44.866590 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.866577    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xdjsz\""
Jul 01 18:06:44.866590 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.866583    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 18:06:44.866803 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.866654    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fvnmh\""
Jul 01 18:06:44.977699 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.977598    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"]
Jul 01 18:06:44.996294 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.996264    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"]
Jul 01 18:06:44.996438 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.996394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:44.998978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.998950    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vkt94\""
Jul 01 18:06:44.999139 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.999023    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 01 18:06:44.999139 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.998953    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 18:06:44.999139 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.998954    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 01 18:06:44.999139 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.998954    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 18:06:44.999139 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:44.998959    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 18:06:45.058197 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.058155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:45.058411 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.058285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:45.058411 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:45.058292    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:06:45.058411 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:45.058388    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm podName:98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:46.058369956 +0000 UTC m=+36.620540585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fjkjl" (UID: "98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34") : configmap "kube-rbac-proxy" not found
Jul 01 18:06:45.058411 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:45.058402    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:06:45.058591 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:45.058456    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls podName:98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:46.058439777 +0000 UTC m=+36.620610406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjkjl" (UID: "98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:06:45.103406 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.103374    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rx2b6" event={"ID":"aafab7ef-f502-4354-a7aa-d42d504684dd","Type":"ContainerStarted","Data":"ac7c5c41ad27728a6426ec3d42dff1de32e589f49974f6e363d534479d51f887"}
Jul 01 18:06:45.104530 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.104506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7mtzl" event={"ID":"89bd1a5c-5b4a-4212-b175-2157b3550774","Type":"ContainerStarted","Data":"84ee81e1794567a5b4d9a77083a0c1ef1ac50e4bac83cd1ec04718d62718f173"}
Jul 01 18:06:45.158926 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.158882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bcd46d5-340a-4a86-8766-1836a6b31198-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.159067 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.158932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9qhs\" (UniqueName: \"kubernetes.io/projected/7bcd46d5-340a-4a86-8766-1836a6b31198-kube-api-access-d9qhs\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.159067 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.159012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bcd46d5-340a-4a86-8766-1836a6b31198-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.159137 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.159121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd46d5-340a-4a86-8766-1836a6b31198-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.259892 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.259809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bcd46d5-340a-4a86-8766-1836a6b31198-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.260346 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.259892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd46d5-340a-4a86-8766-1836a6b31198-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.260346 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.259916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bcd46d5-340a-4a86-8766-1836a6b31198-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.260346 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.259937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9qhs\" (UniqueName: \"kubernetes.io/projected/7bcd46d5-340a-4a86-8766-1836a6b31198-kube-api-access-d9qhs\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.260346 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:45.260042    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 01 18:06:45.260346 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:45.260117    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bcd46d5-340a-4a86-8766-1836a6b31198-prometheus-operator-tls podName:7bcd46d5-340a-4a86-8766-1836a6b31198 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:45.76009939 +0000 UTC m=+36.322270017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/7bcd46d5-340a-4a86-8766-1836a6b31198-prometheus-operator-tls") pod "prometheus-operator-7b7d896497-cbxqd" (UID: "7bcd46d5-340a-4a86-8766-1836a6b31198") : secret "prometheus-operator-tls" not found
Jul 01 18:06:45.260649 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.260605    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd46d5-340a-4a86-8766-1836a6b31198-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.263195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.263176    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bcd46d5-340a-4a86-8766-1836a6b31198-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.270154 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.270133    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9qhs\" (UniqueName: \"kubernetes.io/projected/7bcd46d5-340a-4a86-8766-1836a6b31198-kube-api-access-d9qhs\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.764522 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.764489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bcd46d5-340a-4a86-8766-1836a6b31198-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.767534 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.767499    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bcd46d5-340a-4a86-8766-1836a6b31198-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-cbxqd\" (UID: \"7bcd46d5-340a-4a86-8766-1836a6b31198\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:45.905304 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:45.905268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"
Jul 01 18:06:46.067166 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:46.067123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:46.067326 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:46.067230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:46.067378 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:46.067358    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:06:46.067436 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:46.067422    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls podName:98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:48.067408006 +0000 UTC m=+38.629578621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjkjl" (UID: "98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:06:46.067805 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:46.067789    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:06:46.067876 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:46.067840    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm podName:98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:48.067825932 +0000 UTC m=+38.629996547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fjkjl" (UID: "98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34") : configmap "kube-rbac-proxy" not found
Jul 01 18:06:46.074351 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:46.074325    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-cbxqd"]
Jul 01 18:06:46.078895 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:46.078867    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcd46d5_340a_4a86_8766_1836a6b31198.slice/crio-727cb12db238e251b6cea0cfbbc0831a1b57a4211a7c4b2b150fd0dc3658ce0a WatchSource:0}: Error finding container 727cb12db238e251b6cea0cfbbc0831a1b57a4211a7c4b2b150fd0dc3658ce0a: Status 404 returned error can't find the container with id 727cb12db238e251b6cea0cfbbc0831a1b57a4211a7c4b2b150fd0dc3658ce0a
Jul 01 18:06:46.108699 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:46.108666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd" event={"ID":"7bcd46d5-340a-4a86-8766-1836a6b31198","Type":"ContainerStarted","Data":"727cb12db238e251b6cea0cfbbc0831a1b57a4211a7c4b2b150fd0dc3658ce0a"}
Jul 01 18:06:47.552675 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.552633    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65c6478d64-mgglq"]
Jul 01 18:06:47.562473 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.562439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.568962 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.568935    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 18:06:47.569122 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.568937    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 18:06:47.569122 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.568999    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 18:06:47.569230 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.569171    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 18:06:47.569230 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.569213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 18:06:47.569326 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.569296    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 18:06:47.569365 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.569325    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 18:06:47.569893 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.569869    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rb6k2\""
Jul 01 18:06:47.570577 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.570560    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c6478d64-mgglq"]
Jul 01 18:06:47.680199 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.679884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp4q4\" (UniqueName: \"kubernetes.io/projected/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-kube-api-access-pp4q4\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.680199 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.679937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-oauth-config\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.680199 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.680012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-serving-cert\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.680199 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.680030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-oauth-serving-cert\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.680199 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.680060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-service-ca\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.680199 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.680083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-config\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.781313 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.781269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp4q4\" (UniqueName: \"kubernetes.io/projected/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-kube-api-access-pp4q4\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.781490 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.781328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-oauth-config\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.781490 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.781445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-serving-cert\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.781490 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.781471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-oauth-serving-cert\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.781683 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.781517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-service-ca\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.781683 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.781554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-config\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.782415 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.782392    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-config\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.782520 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.782414    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-oauth-serving-cert\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.782520 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.782395    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-service-ca\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.785285 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.785261    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-oauth-config\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.785401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.785320    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-serving-cert\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.791424 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.791397    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp4q4\" (UniqueName: \"kubernetes.io/projected/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-kube-api-access-pp4q4\") pod \"console-65c6478d64-mgglq\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") " pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:47.877124 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:47.876895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:06:48.084319 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:48.084286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:48.084588 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:48.084407    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:06:48.084588 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:48.084422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:48.084588 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:48.084521    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm podName:98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:52.084483552 +0000 UTC m=+42.646654180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fjkjl" (UID: "98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34") : configmap "kube-rbac-proxy" not found
Jul 01 18:06:48.084588 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:48.084562    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:06:48.084837 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:48.084637    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls podName:98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:52.084601042 +0000 UTC m=+42.646771675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjkjl" (UID: "98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:06:48.410483 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:48.410266    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c6478d64-mgglq"]
Jul 01 18:06:48.415354 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:48.415309    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fdecbf_c6ac_4a1e_84b6_d8e4d6a4c845.slice/crio-3f7d9b4f7da9c31e9e5ee9181b3f55f49b3d1c2fd7e4b0f6d54cbc2b26be8dc3 WatchSource:0}: Error finding container 3f7d9b4f7da9c31e9e5ee9181b3f55f49b3d1c2fd7e4b0f6d54cbc2b26be8dc3: Status 404 returned error can't find the container with id 3f7d9b4f7da9c31e9e5ee9181b3f55f49b3d1c2fd7e4b0f6d54cbc2b26be8dc3
Jul 01 18:06:49.117722 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:49.117657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7mtzl" event={"ID":"89bd1a5c-5b4a-4212-b175-2157b3550774","Type":"ContainerStarted","Data":"0c02d356677b41bac0392cf72e1198bfb00bd7467e1805ef371939916021e123"}
Jul 01 18:06:49.119142 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:49.119111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c6478d64-mgglq" event={"ID":"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845","Type":"ContainerStarted","Data":"3f7d9b4f7da9c31e9e5ee9181b3f55f49b3d1c2fd7e4b0f6d54cbc2b26be8dc3"}
Jul 01 18:06:49.121132 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:49.121106    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rx2b6" event={"ID":"aafab7ef-f502-4354-a7aa-d42d504684dd","Type":"ContainerStarted","Data":"57743e551d89e119e64c253fb7c55279ccae89d43332fd53e5276cec98445d8f"}
Jul 01 18:06:49.121231 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:49.121156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rx2b6" event={"ID":"aafab7ef-f502-4354-a7aa-d42d504684dd","Type":"ContainerStarted","Data":"155f672cc558e956b62a78d68a3655e1b18f7e4c9baf04642dcfd5254808637b"}
Jul 01 18:06:49.121406 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:49.121373    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:49.135555 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:49.135472    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7mtzl" podStartSLOduration=1.7114092250000001 podStartE2EDuration="5.135453292s" totalImagesPullingTime="3.424044067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:49.13456705 +0000 UTC m=+39.696737688" watchObservedRunningTime="2026-07-01 18:06:49.135453292 +0000 UTC m=+39.697623928"
Jul 01 18:06:49.152311 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:49.152260    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rx2b6" podStartSLOduration=1.7673094520000001 podStartE2EDuration="5.152242334s" totalImagesPullingTime="3.384932882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:49.150942547 +0000 UTC m=+39.713113183" watchObservedRunningTime="2026-07-01 18:06:49.152242334 +0000 UTC m=+39.714412970"
Jul 01 18:06:52.122209 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:52.122171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:52.122681 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:52.122258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:06:52.122681 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:52.122348    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:06:52.122681 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:52.122407    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:06:52.122681 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:52.122448    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm podName:98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:00.122419344 +0000 UTC m=+50.684589981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fjkjl" (UID: "98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34") : configmap "kube-rbac-proxy" not found
Jul 01 18:06:52.122681 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:52.122470    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls podName:98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34 nodeName:}" failed. No retries permitted until 2026-07-01 18:07:00.122459912 +0000 UTC m=+50.684630527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjkjl" (UID: "98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:06:53.136404 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:53.136359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd" event={"ID":"7bcd46d5-340a-4a86-8766-1836a6b31198","Type":"ContainerStarted","Data":"7a4f1470577f63ae23a2d380ed10988d847e79d9ef91fd5a0667daa5ccad8ca4"}
Jul 01 18:06:53.136879 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:53.136452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd" event={"ID":"7bcd46d5-340a-4a86-8766-1836a6b31198","Type":"ContainerStarted","Data":"51fc770f900fa435f34e84a04be359543537113a611a11b2a5cfc559225d60eb"}
Jul 01 18:06:53.139116 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:53.139090    2577 generic.go:356] "Generic (PLEG): container finished" podID="79d3fc42-2365-420d-9acd-27da9298db0c" containerID="3d60a198b713eb4e13897624f07333700672da6f5d73b2007adc9b8ab1e093e0" exitCode=0
Jul 01 18:06:53.139255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:53.139132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnxkj" event={"ID":"79d3fc42-2365-420d-9acd-27da9298db0c","Type":"ContainerDied","Data":"3d60a198b713eb4e13897624f07333700672da6f5d73b2007adc9b8ab1e093e0"}
Jul 01 18:06:53.154540 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:53.154498    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-cbxqd" podStartSLOduration=3.068881157 podStartE2EDuration="9.154483947s" totalImagesPullingTime="6.08560279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:53.153031437 +0000 UTC m=+43.715202070" watchObservedRunningTime="2026-07-01 18:06:53.154483947 +0000 UTC m=+43.716654580"
Jul 01 18:06:54.145163 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:54.145131    2577 generic.go:356] "Generic (PLEG): container finished" podID="79d3fc42-2365-420d-9acd-27da9298db0c" containerID="fc2f70fea52fbe3223088132bd59d92b2e471cdd3374df490a7469f1d87780cb" exitCode=0
Jul 01 18:06:54.145567 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:54.145202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnxkj" event={"ID":"79d3fc42-2365-420d-9acd-27da9298db0c","Type":"ContainerDied","Data":"fc2f70fea52fbe3223088132bd59d92b2e471cdd3374df490a7469f1d87780cb"}
Jul 01 18:06:55.152599 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.152563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnxkj" event={"ID":"79d3fc42-2365-420d-9acd-27da9298db0c","Type":"ContainerStarted","Data":"f7456144ee6d88dad3b2577b5a1995aaee81aee77b69ff9f66381b1d3832c3c7"}
Jul 01 18:06:55.176498 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.176451    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vnxkj" podStartSLOduration=7.697145308 podStartE2EDuration="45.176432539s" totalImagesPullingTime="37.479287231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:06:55.17505475 +0000 UTC m=+45.737225386" watchObservedRunningTime="2026-07-01 18:06:55.176432539 +0000 UTC m=+45.738603174"
Jul 01 18:06:55.313478 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.313450    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"]
Jul 01 18:06:55.336762 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.336728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"]
Jul 01 18:06:55.336924 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.336853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.339392 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.339366    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 01 18:06:55.339542 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.339401    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 01 18:06:55.339542 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.339414    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4tmdn\""
Jul 01 18:06:55.344976 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.344956    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hkt6w"]
Jul 01 18:06:55.350160 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.350137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9d4acf-3824-409a-806e-b297e5daeaf8-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.350272 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.350177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e9d4acf-3824-409a-806e-b297e5daeaf8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.350272 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.350229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcmqz\" (UniqueName: \"kubernetes.io/projected/9e9d4acf-3824-409a-806e-b297e5daeaf8-kube-api-access-kcmqz\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.350345 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.350277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e9d4acf-3824-409a-806e-b297e5daeaf8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.368904 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.368877    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-zmrjk"]
Jul 01 18:06:55.369063 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.368999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.371369 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.371345    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 18:06:55.371369 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.371367    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 18:06:55.371564 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.371422    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 18:06:55.371792 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.371775    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7mlgp\""
Jul 01 18:06:55.391072 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.391038    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-zmrjk"]
Jul 01 18:06:55.391181 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.391158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.393381 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.393362    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 01 18:06:55.393535 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.393516    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 01 18:06:55.393637 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.393596    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 01 18:06:55.393788 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.393773    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cdkdm\""
Jul 01 18:06:55.451037 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.450956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-textfile\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.451037 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.450989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-wtmp\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.451037 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.451037 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e9d4acf-3824-409a-806e-b297e5daeaf8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.451288 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.451288 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-accelerators-collector-config\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.451288 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:55.451148    2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 01 18:06:55.451288 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e924e627-b425-4b38-8991-7954dc89ecf8-sys\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.451288 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e9d4acf-3824-409a-806e-b297e5daeaf8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.451288 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:55.451261    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e9d4acf-3824-409a-806e-b297e5daeaf8-openshift-state-metrics-tls podName:9e9d4acf-3824-409a-806e-b297e5daeaf8 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:55.951241816 +0000 UTC m=+46.513412447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9e9d4acf-3824-409a-806e-b297e5daeaf8-openshift-state-metrics-tls") pod "openshift-state-metrics-57f99c8b49-w92fp" (UID: "9e9d4acf-3824-409a-806e-b297e5daeaf8") : secret "openshift-state-metrics-tls" not found
Jul 01 18:06:55.451505 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e924e627-b425-4b38-8991-7954dc89ecf8-root\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.451505 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.451505 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e924e627-b425-4b38-8991-7954dc89ecf8-metrics-client-ca\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.451505 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.451719 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/728eca3a-b50a-46ca-ac9e-62f5fcec1839-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.451719 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-tls\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.451719 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/728eca3a-b50a-46ca-ac9e-62f5fcec1839-metrics-client-ca\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.451719 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dzb\" (UniqueName: \"kubernetes.io/projected/e924e627-b425-4b38-8991-7954dc89ecf8-kube-api-access-k9dzb\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.451719 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcmqz\" (UniqueName: \"kubernetes.io/projected/9e9d4acf-3824-409a-806e-b297e5daeaf8-kube-api-access-kcmqz\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.451719 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9d4acf-3824-409a-806e-b297e5daeaf8-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.451912 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.451727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6mv6\" (UniqueName: \"kubernetes.io/projected/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-api-access-h6mv6\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.452352 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.452332    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9d4acf-3824-409a-806e-b297e5daeaf8-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.455200 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.455167    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e9d4acf-3824-409a-806e-b297e5daeaf8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.463129 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.463104    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcmqz\" (UniqueName: \"kubernetes.io/projected/9e9d4acf-3824-409a-806e-b297e5daeaf8-kube-api-access-kcmqz\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.552690 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.552656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dzb\" (UniqueName: \"kubernetes.io/projected/e924e627-b425-4b38-8991-7954dc89ecf8-kube-api-access-k9dzb\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.552875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.552724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mv6\" (UniqueName: \"kubernetes.io/projected/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-api-access-h6mv6\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.552875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.552787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-textfile\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.552875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.552811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-wtmp\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.552875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.552841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.553075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.552895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.553075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.552920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-accelerators-collector-config\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.552935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e924e627-b425-4b38-8991-7954dc89ecf8-sys\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.552962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e924e627-b425-4b38-8991-7954dc89ecf8-root\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.552988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.553075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e924e627-b425-4b38-8991-7954dc89ecf8-metrics-client-ca\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553068    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e924e627-b425-4b38-8991-7954dc89ecf8-root\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553437 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/728eca3a-b50a-46ca-ac9e-62f5fcec1839-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.553437 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553150    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-textfile\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553437 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-tls\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553437 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/728eca3a-b50a-46ca-ac9e-62f5fcec1839-metrics-client-ca\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.553437 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:55.553238    2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 01 18:06:55.553437 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:55.553288    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-tls podName:728eca3a-b50a-46ca-ac9e-62f5fcec1839 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:56.053272052 +0000 UTC m=+46.615442682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-zmrjk" (UID: "728eca3a-b50a-46ca-ac9e-62f5fcec1839") : secret "kube-state-metrics-tls" not found
Jul 01 18:06:55.553745 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553449    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/728eca3a-b50a-46ca-ac9e-62f5fcec1839-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.553745 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553488    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e924e627-b425-4b38-8991-7954dc89ecf8-sys\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553745 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553503    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-wtmp\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553745 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553694    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e924e627-b425-4b38-8991-7954dc89ecf8-metrics-client-ca\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.553938 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553835    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.553938 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.553924    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/728eca3a-b50a-46ca-ac9e-62f5fcec1839-metrics-client-ca\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.554109 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.554068    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-accelerators-collector-config\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.555906 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.555881    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-tls\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.556224 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.556202    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e924e627-b425-4b38-8991-7954dc89ecf8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.556315 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.556265    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.565169 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.561307    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dzb\" (UniqueName: \"kubernetes.io/projected/e924e627-b425-4b38-8991-7954dc89ecf8-kube-api-access-k9dzb\") pod \"node-exporter-hkt6w\" (UID: \"e924e627-b425-4b38-8991-7954dc89ecf8\") " pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.565169 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.561555    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mv6\" (UniqueName: \"kubernetes.io/projected/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-api-access-h6mv6\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:55.678546 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.678514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hkt6w"
Jul 01 18:06:55.716650 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:55.716595    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode924e627_b425_4b38_8991_7954dc89ecf8.slice/crio-ff35434ac47def4f8e09ccf4915602639498a6a50c8b1866a629a054c597e59f WatchSource:0}: Error finding container ff35434ac47def4f8e09ccf4915602639498a6a50c8b1866a629a054c597e59f: Status 404 returned error can't find the container with id ff35434ac47def4f8e09ccf4915602639498a6a50c8b1866a629a054c597e59f
Jul 01 18:06:55.957214 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.957184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e9d4acf-3824-409a-806e-b297e5daeaf8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:55.959678 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:55.959631    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e9d4acf-3824-409a-806e-b297e5daeaf8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-w92fp\" (UID: \"9e9d4acf-3824-409a-806e-b297e5daeaf8\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:56.057728 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.057690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:56.060302 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.060276    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/728eca3a-b50a-46ca-ac9e-62f5fcec1839-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-zmrjk\" (UID: \"728eca3a-b50a-46ca-ac9e-62f5fcec1839\") " pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:56.157165 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.157134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hkt6w" event={"ID":"e924e627-b425-4b38-8991-7954dc89ecf8","Type":"ContainerStarted","Data":"ff35434ac47def4f8e09ccf4915602639498a6a50c8b1866a629a054c597e59f"}
Jul 01 18:06:56.247652 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.247561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"
Jul 01 18:06:56.300055 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.300016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk"
Jul 01 18:06:56.428630 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.428518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp"]
Jul 01 18:06:56.433468 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.433446    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:06:56.447222 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.447197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.450020 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.450000    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 18:06:56.450228 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.450214    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 18:06:56.451142 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.451119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jdzsf\""
Jul 01 18:06:56.451297 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.451250    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 18:06:56.451297 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.451119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 18:06:56.452328 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.451670    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 18:06:56.452328 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.451866    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 18:06:56.452328 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.451872    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 18:06:56.452328 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.452078    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 18:06:56.452328 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.452161    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 18:06:56.459590 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.459491    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:06:56.461415 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461537 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-web-config\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461537 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461693 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4h5l\" (UniqueName: \"kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-kube-api-access-b4h5l\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461693 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461693 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461693 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461888 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461888 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461888 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-config-out\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461888 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461888 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.461888 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.461881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.507011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.506940    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-zmrjk"]
Jul 01 18:06:56.562777 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.562747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-web-config\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.562921 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.562781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.562921 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.562800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4h5l\" (UniqueName: \"kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-kube-api-access-b4h5l\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.563076 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:56.562913    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 01 18:06:56.563076 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:56.562982    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-main-tls podName:99e99513-6cab-4720-b096-39fd329a09f2 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:57.06296257 +0000 UTC m=+47.625133187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "99e99513-6cab-4720-b096-39fd329a09f2") : secret "alertmanager-main-tls" not found
Jul 01 18:06:56.563076 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.563048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.563076 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.563075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.563273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.563102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.563273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.563156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.563273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.563200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.563273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.563222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-config-out\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.563273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.563243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.563518 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.563288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.563518 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.563315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.563518 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.563351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.564756 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:06:56.564092    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-trusted-ca-bundle podName:99e99513-6cab-4720-b096-39fd329a09f2 nodeName:}" failed. No retries permitted until 2026-07-01 18:06:57.064070937 +0000 UTC m=+47.626241551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "99e99513-6cab-4720-b096-39fd329a09f2") : configmap references non-existent config key: ca-bundle.crt
Jul 01 18:06:56.564756 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.564607    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.564935 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.564876    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.566098 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.565986    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-web-config\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.566433 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.566393    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.566763 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.566713    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.567338 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.567299    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.568194 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.568150    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.568276 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.568261    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.568330 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.568315    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.568867 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.568842    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-config-out\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.573224 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:56.573196    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4h5l\" (UniqueName: \"kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-kube-api-access-b4h5l\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:56.859119 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:56.859085    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728eca3a_b50a_46ca_ac9e_62f5fcec1839.slice/crio-b160483dd83b27fecbdd7b976f047c3a98aa614e655cc7026b03cee1b4e8ed1b WatchSource:0}: Error finding container b160483dd83b27fecbdd7b976f047c3a98aa614e655cc7026b03cee1b4e8ed1b: Status 404 returned error can't find the container with id b160483dd83b27fecbdd7b976f047c3a98aa614e655cc7026b03cee1b4e8ed1b
Jul 01 18:06:57.067034 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.066996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:57.067222 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.067129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:57.067963 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.067936    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:57.069701 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.069678    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:57.098664 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.098605    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:06:57.163819 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.163784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp" event={"ID":"9e9d4acf-3824-409a-806e-b297e5daeaf8","Type":"ContainerStarted","Data":"8c67544afa9f30ceb6f7a76e0a584e371bb520c6d0bd79b0d35aad6285c4c3c5"}
Jul 01 18:06:57.164229 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.163875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp" event={"ID":"9e9d4acf-3824-409a-806e-b297e5daeaf8","Type":"ContainerStarted","Data":"d321ab53d81fa53f31047fa19617e5c9459a606166e35b035e0dd48d38d7fd93"}
Jul 01 18:06:57.164229 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.163909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp" event={"ID":"9e9d4acf-3824-409a-806e-b297e5daeaf8","Type":"ContainerStarted","Data":"0f4ad635f1acca8cc35d8d3972508082d3a925ee4f1113ae2b043fad4eaf8c28"}
Jul 01 18:06:57.164984 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.164957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk" event={"ID":"728eca3a-b50a-46ca-ac9e-62f5fcec1839","Type":"ContainerStarted","Data":"b160483dd83b27fecbdd7b976f047c3a98aa614e655cc7026b03cee1b4e8ed1b"}
Jul 01 18:06:57.383042 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.382989    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:06:57.385152 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:06:57.385127    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e99513_6cab_4720_b096_39fd329a09f2.slice/crio-229ee8551f521ec63fc1d27643a1ae59914426dd3e903ee9fbaa469bde799bc3 WatchSource:0}: Error finding container 229ee8551f521ec63fc1d27643a1ae59914426dd3e903ee9fbaa469bde799bc3: Status 404 returned error can't find the container with id 229ee8551f521ec63fc1d27643a1ae59914426dd3e903ee9fbaa469bde799bc3
Jul 01 18:06:57.936898 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.936866    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dcb68b984-vwjsk"]
Jul 01 18:06:57.942342 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.942313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:57.952092 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.952067    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 01 18:06:57.955041 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.954978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcb68b984-vwjsk"]
Jul 01 18:06:57.977092 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.977056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-serving-cert\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:57.977272 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.977253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-config\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:57.977342 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.977288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-service-ca\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:57.977342 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.977315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddv7\" (UniqueName: \"kubernetes.io/projected/b35752ab-ccd9-41c1-b954-f02d12a85bc3-kube-api-access-9ddv7\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:57.977427 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.977391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-oauth-config\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:57.977468 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.977430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-trusted-ca-bundle\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:57.977468 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:57.977453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-oauth-serving-cert\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.078710 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.078679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-config\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.078710 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.078711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-service-ca\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.078938 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.078732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddv7\" (UniqueName: \"kubernetes.io/projected/b35752ab-ccd9-41c1-b954-f02d12a85bc3-kube-api-access-9ddv7\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.078938 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.078792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-oauth-config\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.078938 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.078822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-trusted-ca-bundle\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.078938 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.078845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-oauth-serving-cert\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.078938 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.078885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-serving-cert\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.079400 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.079376    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-service-ca\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.080028 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.080005    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-config\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.080213 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.080173    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-oauth-serving-cert\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.080725 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.080705    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-trusted-ca-bundle\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.082028 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.082000    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-oauth-config\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.082214 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.082195    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-serving-cert\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.092777 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.092747    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddv7\" (UniqueName: \"kubernetes.io/projected/b35752ab-ccd9-41c1-b954-f02d12a85bc3-kube-api-access-9ddv7\") pod \"console-6dcb68b984-vwjsk\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") " pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.170790 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.170754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerStarted","Data":"229ee8551f521ec63fc1d27643a1ae59914426dd3e903ee9fbaa469bde799bc3"}
Jul 01 18:06:58.172594 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.172559    2577 generic.go:356] "Generic (PLEG): container finished" podID="e924e627-b425-4b38-8991-7954dc89ecf8" containerID="9e9fbe5544c27305388dfadc8e006fa3257494f7ccda8c45c3a0b0997347c9d0" exitCode=0
Jul 01 18:06:58.172741 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.172656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hkt6w" event={"ID":"e924e627-b425-4b38-8991-7954dc89ecf8","Type":"ContainerDied","Data":"9e9fbe5544c27305388dfadc8e006fa3257494f7ccda8c45c3a0b0997347c9d0"}
Jul 01 18:06:58.255583 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.255511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:06:58.310202 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.310171    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6675f8b559-zb88l"]
Jul 01 18:06:58.324357 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.324329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.327545 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.327509    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-31ihbkca6o28f\""
Jul 01 18:06:58.327710 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.327571    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-d8xqp\""
Jul 01 18:06:58.327797 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.327754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 01 18:06:58.327797 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.327771    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 01 18:06:58.327984 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.327966    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 01 18:06:58.328057 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.328037    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 01 18:06:58.328057 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.328050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 01 18:06:58.328932 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.328905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6675f8b559-zb88l"]
Jul 01 18:06:58.381628 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.381581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.381783 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.381696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-grpc-tls\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.381783 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.381733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-tls\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.381783 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.381760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-metrics-client-ca\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.381958 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.381888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.381958 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.381925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c26w\" (UniqueName: \"kubernetes.io/projected/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-kube-api-access-7c26w\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.382059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.381996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.382059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.382029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.483196 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.483158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-tls\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.483379 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.483207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-metrics-client-ca\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.483454 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.483427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.483506 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.483474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c26w\" (UniqueName: \"kubernetes.io/projected/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-kube-api-access-7c26w\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.483571 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.483539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.483646 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.483625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.484183 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.483728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.484183 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.483773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-grpc-tls\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.484183 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.484070    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-metrics-client-ca\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.490661 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.486970    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.490661 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.487033    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-grpc-tls\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.490928 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.490892    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-tls\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.492049 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.492022    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.494491 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.494162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.494491 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.494177    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.495503 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.495482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c26w\" (UniqueName: \"kubernetes.io/projected/a0e787a9-dec5-4d9b-b0f5-cfb65785e82a-kube-api-access-7c26w\") pod \"thanos-querier-6675f8b559-zb88l\" (UID: \"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a\") " pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:58.636700 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:58.636662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:06:59.126995 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.126968    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rx2b6"
Jul 01 18:06:59.616659 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.616602    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9b9977845-hjnlw"]
Jul 01 18:06:59.690566 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.690530    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b9977845-hjnlw"]
Jul 01 18:06:59.690734 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.690655    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.693108 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.693082    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 01 18:06:59.694161 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.694123    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 01 18:06:59.694161 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.694142    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s79ck\""
Jul 01 18:06:59.694361 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.694137    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 01 18:06:59.694361 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.694217    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 18:06:59.694361 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.694137    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4laa7ataai39c\""
Jul 01 18:06:59.795153 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.795119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-audit-log\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.795153 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.795156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-client-ca-bundle\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.795369 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.795237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6msds\" (UniqueName: \"kubernetes.io/projected/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-kube-api-access-6msds\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.795369 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.795266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-secret-metrics-server-tls\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.795369 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.795330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-secret-metrics-server-client-certs\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.795488 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.795408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.795488 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.795463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-metrics-server-audit-profiles\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.896204 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.896135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-metrics-server-audit-profiles\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.896204 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.896170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-audit-log\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.896204 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.896193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-client-ca-bundle\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.896454 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.896347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6msds\" (UniqueName: \"kubernetes.io/projected/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-kube-api-access-6msds\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.896454 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.896403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-secret-metrics-server-tls\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.896454 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.896438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-secret-metrics-server-client-certs\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.896594 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.896497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.896594 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.896570    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-audit-log\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.897140 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.897117    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.897140 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.897138    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-metrics-server-audit-profiles\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.898871 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.898853    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-client-ca-bundle\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.899345 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.899325    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-secret-metrics-server-tls\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.899436 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.899400    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-secret-metrics-server-client-certs\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:06:59.904125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:06:59.904104    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6msds\" (UniqueName: \"kubernetes.io/projected/fb821ca4-9c9e-4fc4-903e-3c932c30bd87-kube-api-access-6msds\") pod \"metrics-server-9b9977845-hjnlw\" (UID: \"fb821ca4-9c9e-4fc4-903e-3c932c30bd87\") " pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:07:00.000677 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.000644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:07:00.095057 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.095023    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"]
Jul 01 18:07:00.114037 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.114011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"]
Jul 01 18:07:00.114203 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.114102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"
Jul 01 18:07:00.116603 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.116575    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 01 18:07:00.116743 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.116626    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mqlrs\""
Jul 01 18:07:00.198656 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.198566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:07:00.198656 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.198608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfwvl\" (UniqueName: \"kubernetes.io/projected/910e51e8-215b-40ab-807d-cf3df2229643-kube-api-access-nfwvl\") pod \"monitoring-plugin-cd8656b8-hm9jk\" (UID: \"910e51e8-215b-40ab-807d-cf3df2229643\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"
Jul 01 18:07:00.198885 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.198707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/910e51e8-215b-40ab-807d-cf3df2229643-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-hm9jk\" (UID: \"910e51e8-215b-40ab-807d-cf3df2229643\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"
Jul 01 18:07:00.198885 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.198778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:07:00.199202 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.199184    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:07:00.201555 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.201530    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjkjl\" (UID: \"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34\") " pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:07:00.206476 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.206447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fjkjl"
Jul 01 18:07:00.299479 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.299440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/910e51e8-215b-40ab-807d-cf3df2229643-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-hm9jk\" (UID: \"910e51e8-215b-40ab-807d-cf3df2229643\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"
Jul 01 18:07:00.299687 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.299569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfwvl\" (UniqueName: \"kubernetes.io/projected/910e51e8-215b-40ab-807d-cf3df2229643-kube-api-access-nfwvl\") pod \"monitoring-plugin-cd8656b8-hm9jk\" (UID: \"910e51e8-215b-40ab-807d-cf3df2229643\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"
Jul 01 18:07:00.302298 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.302262    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/910e51e8-215b-40ab-807d-cf3df2229643-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-hm9jk\" (UID: \"910e51e8-215b-40ab-807d-cf3df2229643\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"
Jul 01 18:07:00.312289 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.312253    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfwvl\" (UniqueName: \"kubernetes.io/projected/910e51e8-215b-40ab-807d-cf3df2229643-kube-api-access-nfwvl\") pod \"monitoring-plugin-cd8656b8-hm9jk\" (UID: \"910e51e8-215b-40ab-807d-cf3df2229643\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"
Jul 01 18:07:00.425008 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.424977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"
Jul 01 18:07:00.538847 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.538473    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d747cb99-5tb2q"]
Jul 01 18:07:00.567208 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.567174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d747cb99-5tb2q"]
Jul 01 18:07:00.567208 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.567332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.578006 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.572521    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 01 18:07:00.578006 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.572850    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 01 18:07:00.578006 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.573052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-77lz4\""
Jul 01 18:07:00.578006 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.573262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 01 18:07:00.578006 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.573858    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 01 18:07:00.578006 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.574105    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 01 18:07:00.583745 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.583134    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 01 18:07:00.605745 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.605581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda60a1f-0aa6-43dd-b53b-28356df77b66-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.605745 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.605656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-telemeter-client-tls\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.605745 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.605707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-secret-telemeter-client\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.606021 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.605843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda60a1f-0aa6-43dd-b53b-28356df77b66-metrics-client-ca\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.606021 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.605924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-federate-client-tls\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.606120 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.606086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda60a1f-0aa6-43dd-b53b-28356df77b66-serving-certs-ca-bundle\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.606173 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.606118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.606237 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.606178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkp6\" (UniqueName: \"kubernetes.io/projected/cda60a1f-0aa6-43dd-b53b-28356df77b66-kube-api-access-jjkp6\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.644789 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.644018    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcb68b984-vwjsk"]
Jul 01 18:07:00.661399 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.661367    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fjkjl"]
Jul 01 18:07:00.672862 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:00.672826    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35752ab_ccd9_41c1_b954_f02d12a85bc3.slice/crio-46ab35f0bc1756a0c031fb9263f4ad0949c6ba96fe8b5d1c5a5fe4c95b14fdc2 WatchSource:0}: Error finding container 46ab35f0bc1756a0c031fb9263f4ad0949c6ba96fe8b5d1c5a5fe4c95b14fdc2: Status 404 returned error can't find the container with id 46ab35f0bc1756a0c031fb9263f4ad0949c6ba96fe8b5d1c5a5fe4c95b14fdc2
Jul 01 18:07:00.673976 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:00.673659    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f2c78e_7b2c_4ab3_8c81_5bfb73ae4d34.slice/crio-e7f4817a64a72440665c638dc71e2a39f90e95f7069b3c6b3202a4dc809b17ff WatchSource:0}: Error finding container e7f4817a64a72440665c638dc71e2a39f90e95f7069b3c6b3202a4dc809b17ff: Status 404 returned error can't find the container with id e7f4817a64a72440665c638dc71e2a39f90e95f7069b3c6b3202a4dc809b17ff
Jul 01 18:07:00.682525 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.682501    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6675f8b559-zb88l"]
Jul 01 18:07:00.683903 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:00.683880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e787a9_dec5_4d9b_b0f5_cfb65785e82a.slice/crio-fda3115924bcaddf249b2bca5db907ee5ef96fb3c601464a306c5b40d8f5177c WatchSource:0}: Error finding container fda3115924bcaddf249b2bca5db907ee5ef96fb3c601464a306c5b40d8f5177c: Status 404 returned error can't find the container with id fda3115924bcaddf249b2bca5db907ee5ef96fb3c601464a306c5b40d8f5177c
Jul 01 18:07:00.708312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.706755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda60a1f-0aa6-43dd-b53b-28356df77b66-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.708312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.706798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-telemeter-client-tls\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.708312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.706837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-secret-telemeter-client\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.708312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.706872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda60a1f-0aa6-43dd-b53b-28356df77b66-metrics-client-ca\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.708312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.706899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-federate-client-tls\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.708312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.706965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda60a1f-0aa6-43dd-b53b-28356df77b66-serving-certs-ca-bundle\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.708312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.706992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.708312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.707021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkp6\" (UniqueName: \"kubernetes.io/projected/cda60a1f-0aa6-43dd-b53b-28356df77b66-kube-api-access-jjkp6\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.708312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.707987    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda60a1f-0aa6-43dd-b53b-28356df77b66-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.708312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.708168    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda60a1f-0aa6-43dd-b53b-28356df77b66-serving-certs-ca-bundle\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.709637 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.709266    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda60a1f-0aa6-43dd-b53b-28356df77b66-metrics-client-ca\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.710914 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.710890    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-secret-telemeter-client\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.711436 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.711412    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.712082 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.712066    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-federate-client-tls\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.712856 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.712835    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cda60a1f-0aa6-43dd-b53b-28356df77b66-telemeter-client-tls\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.720803 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.719573    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkp6\" (UniqueName: \"kubernetes.io/projected/cda60a1f-0aa6-43dd-b53b-28356df77b66-kube-api-access-jjkp6\") pod \"telemeter-client-d747cb99-5tb2q\" (UID: \"cda60a1f-0aa6-43dd-b53b-28356df77b66\") " pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.899366 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.899121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q"
Jul 01 18:07:00.907007 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.906928    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"]
Jul 01 18:07:00.915591 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:00.915520    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b9977845-hjnlw"]
Jul 01 18:07:00.924825 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:00.924796    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb821ca4_9c9e_4fc4_903e_3c932c30bd87.slice/crio-0b257abb27de2d91ff0ee9777b64e64da531f9d74930596327db7db918d42404 WatchSource:0}: Error finding container 0b257abb27de2d91ff0ee9777b64e64da531f9d74930596327db7db918d42404: Status 404 returned error can't find the container with id 0b257abb27de2d91ff0ee9777b64e64da531f9d74930596327db7db918d42404
Jul 01 18:07:01.099482 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.099451    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d747cb99-5tb2q"]
Jul 01 18:07:01.192472 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.191330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hkt6w" event={"ID":"e924e627-b425-4b38-8991-7954dc89ecf8","Type":"ContainerStarted","Data":"9ff23667f1e6e82a9ee9fa2e1359072e390329049c91124edae59cff2f9ccfae"}
Jul 01 18:07:01.192472 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.191394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hkt6w" event={"ID":"e924e627-b425-4b38-8991-7954dc89ecf8","Type":"ContainerStarted","Data":"00213207a7d99a2cbe2249fe24288691ab6bd00a3c4d110507d06471ca5b6427"}
Jul 01 18:07:01.196483 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.196448    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp" event={"ID":"9e9d4acf-3824-409a-806e-b297e5daeaf8","Type":"ContainerStarted","Data":"9f8f7ccadd7a787d9622e346034ae6832bef9e66b0735583e54ee6e63802daa0"}
Jul 01 18:07:01.199404 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.199375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk" event={"ID":"910e51e8-215b-40ab-807d-cf3df2229643","Type":"ContainerStarted","Data":"539dc189a547612581fde08a3eb02a5ea958db683505736b9005cfd77e074027"}
Jul 01 18:07:01.201518 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.201490    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c6478d64-mgglq" event={"ID":"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845","Type":"ContainerStarted","Data":"e0e6dbe3ad27f4f0f42de361ac2fed45fad6cc68a7cf0728ad1b243d346b8ef3"}
Jul 01 18:07:01.203689 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.203638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fjkjl" event={"ID":"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34","Type":"ContainerStarted","Data":"e7f4817a64a72440665c638dc71e2a39f90e95f7069b3c6b3202a4dc809b17ff"}
Jul 01 18:07:01.205302 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.205237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b9977845-hjnlw" event={"ID":"fb821ca4-9c9e-4fc4-903e-3c932c30bd87","Type":"ContainerStarted","Data":"0b257abb27de2d91ff0ee9777b64e64da531f9d74930596327db7db918d42404"}
Jul 01 18:07:01.206593 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.206573    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l" event={"ID":"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a","Type":"ContainerStarted","Data":"fda3115924bcaddf249b2bca5db907ee5ef96fb3c601464a306c5b40d8f5177c"}
Jul 01 18:07:01.208569 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.208497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcb68b984-vwjsk" event={"ID":"b35752ab-ccd9-41c1-b954-f02d12a85bc3","Type":"ContainerStarted","Data":"6f25f17e90967cd91330dcf1b1b0e17859713eeb136da61214f5592d8aef350e"}
Jul 01 18:07:01.208569 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.208542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcb68b984-vwjsk" event={"ID":"b35752ab-ccd9-41c1-b954-f02d12a85bc3","Type":"ContainerStarted","Data":"46ab35f0bc1756a0c031fb9263f4ad0949c6ba96fe8b5d1c5a5fe4c95b14fdc2"}
Jul 01 18:07:01.210696 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.209961    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hkt6w" podStartSLOduration=4.570959237 podStartE2EDuration="6.209943994s" totalImagesPullingTime="1.522323914s" totalInitContainerRuntime="116.660843ms" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:01.209580467 +0000 UTC m=+51.771751103" watchObservedRunningTime="2026-07-01 18:07:01.209943994 +0000 UTC m=+51.772114632"
Jul 01 18:07:01.216261 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.215832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk" event={"ID":"728eca3a-b50a-46ca-ac9e-62f5fcec1839","Type":"ContainerStarted","Data":"b9841210aaafaf69915070bb7dd4a471e254624bf1ff7c030d87406a17d03db9"}
Jul 01 18:07:01.216261 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.215892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk" event={"ID":"728eca3a-b50a-46ca-ac9e-62f5fcec1839","Type":"ContainerStarted","Data":"402f44e910510159ab9620def0d801c73381c261453d67feb5aaf6bae87922da"}
Jul 01 18:07:01.216261 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.215919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk" event={"ID":"728eca3a-b50a-46ca-ac9e-62f5fcec1839","Type":"ContainerStarted","Data":"92262fe66e84400adf8ba74519193a0e7561d9c91bca9891c7a8bbdf933f1561"}
Jul 01 18:07:01.219736 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.219172    2577 generic.go:356] "Generic (PLEG): container finished" podID="99e99513-6cab-4720-b096-39fd329a09f2" containerID="101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92" exitCode=0
Jul 01 18:07:01.219736 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.219459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerDied","Data":"101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92"}
Jul 01 18:07:01.223191 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.223126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q" event={"ID":"cda60a1f-0aa6-43dd-b53b-28356df77b66","Type":"ContainerStarted","Data":"55e9b0fabb3132a93cc0d7b6a21e02f45e69a7c2e41f6e427202b4d56a9c7473"}
Jul 01 18:07:01.250781 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.250507    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dcb68b984-vwjsk" podStartSLOduration=4.250483328 podStartE2EDuration="4.250483328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:01.248446824 +0000 UTC m=+51.810617463" watchObservedRunningTime="2026-07-01 18:07:01.250483328 +0000 UTC m=+51.812653961"
Jul 01 18:07:01.275794 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.275302    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65c6478d64-mgglq" podStartSLOduration=2.242922139 podStartE2EDuration="14.275279301s" totalImagesPullingTime="12.032357162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:01.273965799 +0000 UTC m=+51.836136435" watchObservedRunningTime="2026-07-01 18:07:01.275279301 +0000 UTC m=+51.837449939"
Jul 01 18:07:01.294423 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.294339    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-w92fp" podStartSLOduration=2.81663772 podStartE2EDuration="6.294315775s" totalImagesPullingTime="3.477678055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:01.291778196 +0000 UTC m=+51.853948859" watchObservedRunningTime="2026-07-01 18:07:01.294315775 +0000 UTC m=+51.856486412"
Jul 01 18:07:01.313741 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.313681    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-zmrjk" podStartSLOduration=2.729546331 podStartE2EDuration="6.313660637s" totalImagesPullingTime="3.584114306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:01.312690235 +0000 UTC m=+51.874860873" watchObservedRunningTime="2026-07-01 18:07:01.313660637 +0000 UTC m=+51.875831280"
Jul 01 18:07:01.658391 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.657992    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:07:01.693743 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.685442    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:07:01.693743 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.685581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.695163 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695024    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 18:07:01.695163 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695076    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 18:07:01.695163 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695024    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 18:07:01.695397 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695204    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6fjrtgm03bb4v\""
Jul 01 18:07:01.695397 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 18:07:01.695397 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695286    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wxjd5\""
Jul 01 18:07:01.695606 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695432    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 18:07:01.695606 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695446    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 18:07:01.695606 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695582    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 18:07:01.695606 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695591    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 18:07:01.695897 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.695831    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 18:07:01.696182 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.696037    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 18:07:01.696182 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.696148    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 18:07:01.696556 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.696439    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 18:07:01.699758 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.698847    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 18:07:01.709798 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.709772    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 18:07:01.738125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.737781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.738125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.737828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.738125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.737868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.738125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.737895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.738125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.737920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.738125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.737951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.738125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.737976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.738125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.738125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-config-out\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.738125 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.741795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgts\" (UniqueName: \"kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-kube-api-access-jvgts\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.741795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.741795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.741795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.741795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.741795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.741795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.741795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-web-config\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.741795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.741795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.738551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-config\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.839724 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.839683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.839919 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.839760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-config\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.839919 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.839806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.839919 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.839832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.839919 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.839861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.839919 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.839885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.839919 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.839916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840227 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.839947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840227 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.839973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840227 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840227 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-config-out\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840227 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840227 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgts\" (UniqueName: \"kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-kube-api-access-jvgts\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840227 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840227 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840600 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840600 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840600 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840600 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840600 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-web-config\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.840975 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.840769    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.843062 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.843038    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.844549 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.844466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.844669 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.844496    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.845765 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.845322    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.845765 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.845723    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.846096 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.846050    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.849720 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.849696    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.850940 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.850912    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgts\" (UniqueName: \"kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-kube-api-access-jvgts\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.852915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.852890    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.857180 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.856582    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.857180 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.856668    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.857180 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.856940    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.857180 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.857140    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.857702 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.857412    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-config\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.857702 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.857430    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.859273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.858817    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-config-out\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.859874 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.859849    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.861875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.861277    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-web-config\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:01.861875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:01.861752    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:02.012438 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:02.012366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:02.162480 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:02.162444    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65c6478d64-mgglq"]
Jul 01 18:07:02.217609 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:02.217571    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:07:02.222371 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:02.222337    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7471ca64_952b_4347_bcfe_7565bef0d03a.slice/crio-108a139cf5f63574d8e9a21b0e876ad8218c00151e28125328cc1d561d482da6 WatchSource:0}: Error finding container 108a139cf5f63574d8e9a21b0e876ad8218c00151e28125328cc1d561d482da6: Status 404 returned error can't find the container with id 108a139cf5f63574d8e9a21b0e876ad8218c00151e28125328cc1d561d482da6
Jul 01 18:07:02.238759 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:02.234151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerStarted","Data":"108a139cf5f63574d8e9a21b0e876ad8218c00151e28125328cc1d561d482da6"}
Jul 01 18:07:03.240781 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:03.240738    2577 generic.go:356] "Generic (PLEG): container finished" podID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerID="4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66" exitCode=0
Jul 01 18:07:03.241200 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:03.240825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerDied","Data":"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"}
Jul 01 18:07:07.877881 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:07.877849    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:07:08.255885 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:08.255799    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:07:08.255885 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:08.255863    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:07:08.261221 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:08.261198    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:07:08.266540 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:08.266513    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:07:09.270961 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.270921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fjkjl" event={"ID":"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34","Type":"ContainerStarted","Data":"d13a4aab154e97c0a022d59a12d6c3444e0e6b441dbb7b7440149df55e84002d"}
Jul 01 18:07:09.272495 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.272448    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b9977845-hjnlw" event={"ID":"fb821ca4-9c9e-4fc4-903e-3c932c30bd87","Type":"ContainerStarted","Data":"969d6fec3d87e1b68a5e62c69aca3e54c39920e747aad87cd0913c3040ce66cd"}
Jul 01 18:07:09.276855 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.276733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l" event={"ID":"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a","Type":"ContainerStarted","Data":"3b150e0bcb85e94849417d7b60dbee5ac76aa106f23f7c80094263c47fa5ee2f"}
Jul 01 18:07:09.276855 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.276799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l" event={"ID":"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a","Type":"ContainerStarted","Data":"71b1db40497c077b40b53a4d96beee0872ce750fbc821d161962dc35d9662b8a"}
Jul 01 18:07:09.276855 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.276826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l" event={"ID":"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a","Type":"ContainerStarted","Data":"5615e364e07de79a728511735071ea2b67286e62cdb9b485fca043fb0b98c8e7"}
Jul 01 18:07:09.279419 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.279253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerStarted","Data":"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"}
Jul 01 18:07:09.279419 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.279302    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerStarted","Data":"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"}
Jul 01 18:07:09.279419 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.279326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerStarted","Data":"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"}
Jul 01 18:07:09.279419 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.279350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerStarted","Data":"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"}
Jul 01 18:07:09.279419 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.279378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerStarted","Data":"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"}
Jul 01 18:07:09.282183 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.282144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerStarted","Data":"fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2"}
Jul 01 18:07:09.282253 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.282192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerStarted","Data":"e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93"}
Jul 01 18:07:09.282253 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.282218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerStarted","Data":"2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732"}
Jul 01 18:07:09.282253 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.282241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerStarted","Data":"ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542"}
Jul 01 18:07:09.282428 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.282265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerStarted","Data":"f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0"}
Jul 01 18:07:09.285310 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.285227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q" event={"ID":"cda60a1f-0aa6-43dd-b53b-28356df77b66","Type":"ContainerStarted","Data":"ef4ddb074774a2cb7d3598c729e51087539fcee2ec243654d1c1e5f1fd06e909"}
Jul 01 18:07:09.285310 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.285269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q" event={"ID":"cda60a1f-0aa6-43dd-b53b-28356df77b66","Type":"ContainerStarted","Data":"f34eba3a51e8d7a8d8cf00d7910784fde90d81f9057312ad9467524e1ef9e607"}
Jul 01 18:07:09.285310 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.285288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q" event={"ID":"cda60a1f-0aa6-43dd-b53b-28356df77b66","Type":"ContainerStarted","Data":"75457006143eae8c3dd2ff9cced80a06d95d4748ba3eafbf0cfc44937ae68d09"}
Jul 01 18:07:09.292315 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.291925    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9b9977845-hjnlw" podStartSLOduration=2.605787185 podStartE2EDuration="10.291903633s" totalImagesPullingTime="7.686116448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:09.289669524 +0000 UTC m=+59.851840160" watchObservedRunningTime="2026-07-01 18:07:09.291903633 +0000 UTC m=+59.854074270"
Jul 01 18:07:09.313232 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:09.313134    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d747cb99-5tb2q" podStartSLOduration=1.823109091 podStartE2EDuration="9.313113899s" totalImagesPullingTime="7.490004808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:09.310973659 +0000 UTC m=+59.873144291" watchObservedRunningTime="2026-07-01 18:07:09.313113899 +0000 UTC m=+59.875284536"
Jul 01 18:07:10.102793 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:10.102764    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-44jxb"
Jul 01 18:07:10.293397 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:10.293343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerStarted","Data":"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"}
Jul 01 18:07:10.295018 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:10.294987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk" event={"ID":"910e51e8-215b-40ab-807d-cf3df2229643","Type":"ContainerStarted","Data":"515c8d47b75a30f773656ab1b1a4c0e6c317690d1ad46aaabd620d1526398bb5"}
Jul 01 18:07:10.295168 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:10.295153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"
Jul 01 18:07:10.300435 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:10.300409    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk"
Jul 01 18:07:10.323131 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:10.322975    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.923314284 podStartE2EDuration="9.322953696s" totalImagesPullingTime="5.369212926s" totalInitContainerRuntime="30.426486ms" isStatefulPod=false podCreationTimestamp="2026-07-01 18:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:10.322513793 +0000 UTC m=+60.884684493" watchObservedRunningTime="2026-07-01 18:07:10.322953696 +0000 UTC m=+60.885124332"
Jul 01 18:07:10.338841 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:10.338773    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-cd8656b8-hm9jk" podStartSLOduration=1.434080219 podStartE2EDuration="10.338757697s" totalImagesPullingTime="8.904677478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:10.338225172 +0000 UTC m=+60.900395821" watchObservedRunningTime="2026-07-01 18:07:10.338757697 +0000 UTC m=+60.900928333"
Jul 01 18:07:11.303516 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:11.303475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerStarted","Data":"c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2"}
Jul 01 18:07:11.327881 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:11.327814    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.995462627 podStartE2EDuration="15.327789983s" totalImagesPullingTime="11.299220702s" totalInitContainerRuntime="33.106654ms" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:11.326991402 +0000 UTC m=+61.889162075" watchObservedRunningTime="2026-07-01 18:07:11.327789983 +0000 UTC m=+61.889960620"
Jul 01 18:07:12.013439 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:12.013351    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:12.310744 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:12.310657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fjkjl" event={"ID":"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34","Type":"ContainerStarted","Data":"f0420f88141083e3687f0f76081c8f134bbe397196cd4c835ec49f57dd5851ea"}
Jul 01 18:07:12.314277 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:12.314246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l" event={"ID":"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a","Type":"ContainerStarted","Data":"eb88919f067f4fd000a1501b3db07f81b8dcbc971f1174979eced91740706e16"}
Jul 01 18:07:12.314414 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:12.314324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l" event={"ID":"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a","Type":"ContainerStarted","Data":"9ae4f57f21d76a18a026ea80bfc3ea8e2b72e44200775c6647edb64f1a46fe7f"}
Jul 01 18:07:12.314414 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:12.314372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l" event={"ID":"a0e787a9-dec5-4d9b-b0f5-cfb65785e82a","Type":"ContainerStarted","Data":"0ff9a3425072917cafe88d71ffcb445ce8a03198f2714aa3b5676048c02783ad"}
Jul 01 18:07:12.316386 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:12.316355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:07:12.859528 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:12.859467    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l" podStartSLOduration=4.484320798 podStartE2EDuration="14.859446343s" totalImagesPullingTime="10.375125545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:12.336735862 +0000 UTC m=+62.898906502" watchObservedRunningTime="2026-07-01 18:07:12.859446343 +0000 UTC m=+63.421616979"
Jul 01 18:07:12.860096 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:12.860078    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcb68b984-vwjsk"]
Jul 01 18:07:13.324914 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:13.324843    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6675f8b559-zb88l"
Jul 01 18:07:14.329877 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:14.329778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fjkjl" event={"ID":"98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34","Type":"ContainerStarted","Data":"a53d23fa375a4991d371353be91a2aa2c6c8a9ea4228e1945f4c44a0dbc50558"}
Jul 01 18:07:14.347882 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:14.347825    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fjkjl" podStartSLOduration=17.178629413 podStartE2EDuration="30.347808455s" totalImagesPullingTime="13.169179042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:14.346389774 +0000 UTC m=+64.908560412" watchObservedRunningTime="2026-07-01 18:07:14.347808455 +0000 UTC m=+64.909979128"
Jul 01 18:07:15.704235 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.704181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:07:15.704724 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.704284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vxl\" (UniqueName: \"kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl\") pod \"network-check-target-stnmw\" (UID: \"51bddbdc-6a46-4d8c-a6ea-f92883c6189a\") " pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:07:15.704724 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.704323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:07:15.707040 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.707022    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 18:07:15.708021 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.708007    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 18:07:15.708059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.708021    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 18:07:15.717503 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.717473    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b74b41-8fc3-4992-9a5a-f3f01dfde169-metrics-certs\") pod \"network-metrics-daemon-qkfdw\" (UID: \"51b74b41-8fc3-4992-9a5a-f3f01dfde169\") " pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:07:15.717635 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.717514    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 18:07:15.717635 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.717538    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787-original-pull-secret\") pod \"global-pull-secret-syncer-dkxdw\" (UID: \"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787\") " pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:07:15.727741 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.727714    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vxl\" (UniqueName: \"kubernetes.io/projected/51bddbdc-6a46-4d8c-a6ea-f92883c6189a-kube-api-access-z6vxl\") pod \"network-check-target-stnmw\" (UID: \"51bddbdc-6a46-4d8c-a6ea-f92883c6189a\") " pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:07:15.774900 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.774866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dkxdw"
Jul 01 18:07:15.783100 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.783074    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xdjsz\""
Jul 01 18:07:15.787286 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.787261    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fvnmh\""
Jul 01 18:07:15.791192 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.791172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:07:15.795174 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.795152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkfdw"
Jul 01 18:07:15.933327 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:15.933294    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dkxdw"]
Jul 01 18:07:15.937377 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:15.937346    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e574eeb_3e4c_4e1c_9e54_9b1bf45f2787.slice/crio-a2f72531aece076c886850453dacd011395cb23c75a94e3a4b8439432454ce41 WatchSource:0}: Error finding container a2f72531aece076c886850453dacd011395cb23c75a94e3a4b8439432454ce41: Status 404 returned error can't find the container with id a2f72531aece076c886850453dacd011395cb23c75a94e3a4b8439432454ce41
Jul 01 18:07:16.157759 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.157674    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qkfdw"]
Jul 01 18:07:16.160828 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:16.160801    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b74b41_8fc3_4992_9a5a_f3f01dfde169.slice/crio-70021788b9c9d5181e5038b8a05d564431327b3011b85b618cf00d069cd79e07 WatchSource:0}: Error finding container 70021788b9c9d5181e5038b8a05d564431327b3011b85b618cf00d069cd79e07: Status 404 returned error can't find the container with id 70021788b9c9d5181e5038b8a05d564431327b3011b85b618cf00d069cd79e07
Jul 01 18:07:16.161062 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.161041    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-stnmw"]
Jul 01 18:07:16.163675 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:16.163656    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bddbdc_6a46_4d8c_a6ea_f92883c6189a.slice/crio-e3afc50a56174003c8aef8b32dfaf0289b76581f824a8f97989b093ae4c8a9f3 WatchSource:0}: Error finding container e3afc50a56174003c8aef8b32dfaf0289b76581f824a8f97989b093ae4c8a9f3: Status 404 returned error can't find the container with id e3afc50a56174003c8aef8b32dfaf0289b76581f824a8f97989b093ae4c8a9f3
Jul 01 18:07:16.340336 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.340289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dkxdw" event={"ID":"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787","Type":"ContainerStarted","Data":"a2f72531aece076c886850453dacd011395cb23c75a94e3a4b8439432454ce41"}
Jul 01 18:07:16.341794 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.341766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-stnmw" event={"ID":"51bddbdc-6a46-4d8c-a6ea-f92883c6189a","Type":"ContainerStarted","Data":"e3afc50a56174003c8aef8b32dfaf0289b76581f824a8f97989b093ae4c8a9f3"}
Jul 01 18:07:16.343019 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.342997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qkfdw" event={"ID":"51b74b41-8fc3-4992-9a5a-f3f01dfde169","Type":"ContainerStarted","Data":"70021788b9c9d5181e5038b8a05d564431327b3011b85b618cf00d069cd79e07"}
Jul 01 18:07:16.406606 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.406565    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cf6b987cc-kl2tp"]
Jul 01 18:07:16.410310 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.410261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.411606 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.411579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-oauth-serving-cert\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.411735 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.411638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-service-ca\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.411735 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.411699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct8pz\" (UniqueName: \"kubernetes.io/projected/13481e8d-918a-4482-b6a5-21d73329b2f6-kube-api-access-ct8pz\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.411803 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.411745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-serving-cert\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.411803 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.411786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-console-config\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.411885 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.411869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-oauth-config\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.411948 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.411929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-trusted-ca-bundle\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.420358 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.420332    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cf6b987cc-kl2tp"]
Jul 01 18:07:16.513079 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.513047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-oauth-config\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.513234 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.513100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-trusted-ca-bundle\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.513234 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.513162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-oauth-serving-cert\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.513234 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.513185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-service-ca\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.513234 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.513221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct8pz\" (UniqueName: \"kubernetes.io/projected/13481e8d-918a-4482-b6a5-21d73329b2f6-kube-api-access-ct8pz\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.513434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.513256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-serving-cert\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.513434 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.513276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-console-config\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.514121 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.514093    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-oauth-serving-cert\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.514262 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.514093    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-service-ca\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.514262 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.514197    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-console-config\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.514411 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.514366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-trusted-ca-bundle\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.515998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.515951    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-oauth-config\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.517093 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.517068    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-serving-cert\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.522195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.522167    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct8pz\" (UniqueName: \"kubernetes.io/projected/13481e8d-918a-4482-b6a5-21d73329b2f6-kube-api-access-ct8pz\") pod \"console-5cf6b987cc-kl2tp\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") " pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.723457 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.723173    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:16.906274 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:16.906233    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cf6b987cc-kl2tp"]
Jul 01 18:07:16.911151 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:16.911116    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13481e8d_918a_4482_b6a5_21d73329b2f6.slice/crio-fb08c7832c3a67e94293e6fef7e16819757785fe946801c438c3d767534b479d WatchSource:0}: Error finding container fb08c7832c3a67e94293e6fef7e16819757785fe946801c438c3d767534b479d: Status 404 returned error can't find the container with id fb08c7832c3a67e94293e6fef7e16819757785fe946801c438c3d767534b479d
Jul 01 18:07:17.352339 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:17.351747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf6b987cc-kl2tp" event={"ID":"13481e8d-918a-4482-b6a5-21d73329b2f6","Type":"ContainerStarted","Data":"a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466"}
Jul 01 18:07:17.352339 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:17.351811    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf6b987cc-kl2tp" event={"ID":"13481e8d-918a-4482-b6a5-21d73329b2f6","Type":"ContainerStarted","Data":"fb08c7832c3a67e94293e6fef7e16819757785fe946801c438c3d767534b479d"}
Jul 01 18:07:17.374117 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:17.371223    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cf6b987cc-kl2tp" podStartSLOduration=1.371198139 podStartE2EDuration="1.371198139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:17.37018848 +0000 UTC m=+67.932359115" watchObservedRunningTime="2026-07-01 18:07:17.371198139 +0000 UTC m=+67.933368776"
Jul 01 18:07:20.000907 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:20.000853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:07:20.001274 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:20.000929    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:07:22.381321 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:22.381288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-stnmw" event={"ID":"51bddbdc-6a46-4d8c-a6ea-f92883c6189a","Type":"ContainerStarted","Data":"9045f817ca01f3cc96a742a0bd18cf47d47824c77148470ae3730611c471aa0d"}
Jul 01 18:07:22.381800 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:22.381449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:07:22.383099 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:22.383074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qkfdw" event={"ID":"51b74b41-8fc3-4992-9a5a-f3f01dfde169","Type":"ContainerStarted","Data":"42708281d5a64ce2ea0e5bdfaf640123872a2de45a13b29cf912af50d0914173"}
Jul 01 18:07:22.384858 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:22.384828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dkxdw" event={"ID":"8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787","Type":"ContainerStarted","Data":"178bde1fcd5b143fa855bea68ffaee4e5e837a7b967a4c57c6225e2e06fc142d"}
Jul 01 18:07:22.398058 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:22.398009    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-stnmw" podStartSLOduration=67.393737329 podStartE2EDuration="1m13.397993062s" totalImagesPullingTime="6.004255733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:22.396276727 +0000 UTC m=+72.958447366" watchObservedRunningTime="2026-07-01 18:07:22.397993062 +0000 UTC m=+72.960163698"
Jul 01 18:07:23.391014 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:23.390976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qkfdw" event={"ID":"51b74b41-8fc3-4992-9a5a-f3f01dfde169","Type":"ContainerStarted","Data":"f9e42d85194337de12f8c6a5a27be5d933f070ceda21e97173cd05fe24c02d4c"}
Jul 01 18:07:23.408207 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:23.408155    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dkxdw" podStartSLOduration=67.165622421 podStartE2EDuration="1m13.408139826s" totalImagesPullingTime="6.242517405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:22.411261696 +0000 UTC m=+72.973432332" watchObservedRunningTime="2026-07-01 18:07:23.408139826 +0000 UTC m=+73.970310465"
Jul 01 18:07:23.408962 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:23.408933    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qkfdw" podStartSLOduration=67.40204045 podStartE2EDuration="1m13.408923444s" totalImagesPullingTime="6.006882994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:23.407497983 +0000 UTC m=+73.969668618" watchObservedRunningTime="2026-07-01 18:07:23.408923444 +0000 UTC m=+73.971094080"
Jul 01 18:07:26.724331 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:26.724289    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:26.724816 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:26.724355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:26.729234 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:26.729209    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:27.421875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:27.421846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:07:28.267648 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.267565    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-65c6478d64-mgglq" podUID="a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" containerName="console" containerID="cri-o://e0e6dbe3ad27f4f0f42de361ac2fed45fad6cc68a7cf0728ad1b243d346b8ef3" gracePeriod=15
Jul 01 18:07:28.424469 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.424437    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" containerID="e0e6dbe3ad27f4f0f42de361ac2fed45fad6cc68a7cf0728ad1b243d346b8ef3" exitCode=0
Jul 01 18:07:28.424666 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.424514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c6478d64-mgglq" event={"ID":"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845","Type":"ContainerDied","Data":"e0e6dbe3ad27f4f0f42de361ac2fed45fad6cc68a7cf0728ad1b243d346b8ef3"}
Jul 01 18:07:28.519988 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.519934    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:07:28.647138 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.647107    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-service-ca\") pod \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") "
Jul 01 18:07:28.647307 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.647190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-oauth-serving-cert\") pod \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") "
Jul 01 18:07:28.647836 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.647556    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-serving-cert\") pod \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") "
Jul 01 18:07:28.647836 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.647762    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-service-ca" (OuterVolumeSpecName: "service-ca") pod "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" (UID: "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:28.647836 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.647814    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-config\") pod \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") "
Jul 01 18:07:28.648951 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.648100    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-oauth-config\") pod \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") "
Jul 01 18:07:28.648951 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.648181    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp4q4\" (UniqueName: \"kubernetes.io/projected/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-kube-api-access-pp4q4\") pod \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\" (UID: \"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845\") "
Jul 01 18:07:28.648951 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.647814    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" (UID: "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:28.648951 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.648241    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-config" (OuterVolumeSpecName: "console-config") pod "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" (UID: "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:28.648951 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.648648    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-oauth-serving-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:28.648951 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.648666    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:28.648951 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.648680    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-service-ca\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:28.651745 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.651140    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" (UID: "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:28.654366 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.654334    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-kube-api-access-pp4q4" (OuterVolumeSpecName: "kube-api-access-pp4q4") pod "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" (UID: "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845"). InnerVolumeSpecName "kube-api-access-pp4q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:07:28.654366 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.654351    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" (UID: "a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:28.749957 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.749916    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp4q4\" (UniqueName: \"kubernetes.io/projected/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-kube-api-access-pp4q4\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:28.749957 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.749950    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-serving-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:28.749957 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:28.749963    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845-console-oauth-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:29.430184 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:29.430153    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c6478d64-mgglq"
Jul 01 18:07:29.430184 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:29.430176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c6478d64-mgglq" event={"ID":"a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845","Type":"ContainerDied","Data":"3f7d9b4f7da9c31e9e5ee9181b3f55f49b3d1c2fd7e4b0f6d54cbc2b26be8dc3"}
Jul 01 18:07:29.430788 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:29.430248    2577 scope.go:117] "RemoveContainer" containerID="e0e6dbe3ad27f4f0f42de361ac2fed45fad6cc68a7cf0728ad1b243d346b8ef3"
Jul 01 18:07:29.451221 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:29.451199    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65c6478d64-mgglq"]
Jul 01 18:07:29.454473 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:29.454450    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65c6478d64-mgglq"]
Jul 01 18:07:29.869050 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:29.869019    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" path="/var/lib/kubelet/pods/a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845/volumes"
Jul 01 18:07:36.172152 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:36.172107    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:36.192141 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:36.192118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:36.476567 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:36.476494    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:37.888261 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:37.888227    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6dcb68b984-vwjsk" podUID="b35752ab-ccd9-41c1-b954-f02d12a85bc3" containerName="console" containerID="cri-o://6f25f17e90967cd91330dcf1b1b0e17859713eeb136da61214f5592d8aef350e" gracePeriod=15
Jul 01 18:07:38.262722 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.262601    2577 patch_prober.go:28] interesting pod/console-6dcb68b984-vwjsk container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused" start-of-body=
Jul 01 18:07:38.262722 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.262690    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6dcb68b984-vwjsk" podUID="b35752ab-ccd9-41c1-b954-f02d12a85bc3" containerName="console" probeResult="failure" output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused"
Jul 01 18:07:38.468767 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.468730    2577 generic.go:356] "Generic (PLEG): container finished" podID="b35752ab-ccd9-41c1-b954-f02d12a85bc3" containerID="6f25f17e90967cd91330dcf1b1b0e17859713eeb136da61214f5592d8aef350e" exitCode=0
Jul 01 18:07:38.468930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.468801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcb68b984-vwjsk" event={"ID":"b35752ab-ccd9-41c1-b954-f02d12a85bc3","Type":"ContainerDied","Data":"6f25f17e90967cd91330dcf1b1b0e17859713eeb136da61214f5592d8aef350e"}
Jul 01 18:07:38.656189 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.656164    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:07:38.742437 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.742396    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-oauth-serving-cert\") pod \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") "
Jul 01 18:07:38.742608 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.742486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-serving-cert\") pod \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") "
Jul 01 18:07:38.742608 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.742507    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-service-ca\") pod \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") "
Jul 01 18:07:38.742608 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.742532    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-oauth-config\") pod \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") "
Jul 01 18:07:38.742608 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.742554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-trusted-ca-bundle\") pod \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") "
Jul 01 18:07:38.742608 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.742587    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-config\") pod \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") "
Jul 01 18:07:38.743120 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.742720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ddv7\" (UniqueName: \"kubernetes.io/projected/b35752ab-ccd9-41c1-b954-f02d12a85bc3-kube-api-access-9ddv7\") pod \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\" (UID: \"b35752ab-ccd9-41c1-b954-f02d12a85bc3\") "
Jul 01 18:07:38.743223 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.742948    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-service-ca" (OuterVolumeSpecName: "service-ca") pod "b35752ab-ccd9-41c1-b954-f02d12a85bc3" (UID: "b35752ab-ccd9-41c1-b954-f02d12a85bc3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:38.743223 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.742970    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b35752ab-ccd9-41c1-b954-f02d12a85bc3" (UID: "b35752ab-ccd9-41c1-b954-f02d12a85bc3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:38.743223 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.743021    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-config" (OuterVolumeSpecName: "console-config") pod "b35752ab-ccd9-41c1-b954-f02d12a85bc3" (UID: "b35752ab-ccd9-41c1-b954-f02d12a85bc3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:38.743223 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.743087    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b35752ab-ccd9-41c1-b954-f02d12a85bc3" (UID: "b35752ab-ccd9-41c1-b954-f02d12a85bc3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:38.743565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.743389    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-oauth-serving-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:38.743565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.743406    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-service-ca\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:38.743565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.743420    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-trusted-ca-bundle\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:38.743565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.743433    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:38.745133 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.745108    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35752ab-ccd9-41c1-b954-f02d12a85bc3-kube-api-access-9ddv7" (OuterVolumeSpecName: "kube-api-access-9ddv7") pod "b35752ab-ccd9-41c1-b954-f02d12a85bc3" (UID: "b35752ab-ccd9-41c1-b954-f02d12a85bc3"). InnerVolumeSpecName "kube-api-access-9ddv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:07:38.745474 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.745446    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b35752ab-ccd9-41c1-b954-f02d12a85bc3" (UID: "b35752ab-ccd9-41c1-b954-f02d12a85bc3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:38.745594 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.745471    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b35752ab-ccd9-41c1-b954-f02d12a85bc3" (UID: "b35752ab-ccd9-41c1-b954-f02d12a85bc3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:38.844197 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.844114    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-serving-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:38.844197 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.844147    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b35752ab-ccd9-41c1-b954-f02d12a85bc3-console-oauth-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:38.844197 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:38.844157    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ddv7\" (UniqueName: \"kubernetes.io/projected/b35752ab-ccd9-41c1-b954-f02d12a85bc3-kube-api-access-9ddv7\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:39.475219 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:39.475187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcb68b984-vwjsk" event={"ID":"b35752ab-ccd9-41c1-b954-f02d12a85bc3","Type":"ContainerDied","Data":"46ab35f0bc1756a0c031fb9263f4ad0949c6ba96fe8b5d1c5a5fe4c95b14fdc2"}
Jul 01 18:07:39.475219 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:39.475213    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcb68b984-vwjsk"
Jul 01 18:07:39.475689 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:39.475245    2577 scope.go:117] "RemoveContainer" containerID="6f25f17e90967cd91330dcf1b1b0e17859713eeb136da61214f5592d8aef350e"
Jul 01 18:07:39.498772 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:39.498747    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcb68b984-vwjsk"]
Jul 01 18:07:39.502270 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:39.502244    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dcb68b984-vwjsk"]
Jul 01 18:07:39.869277 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:39.869242    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b35752ab-ccd9-41c1-b954-f02d12a85bc3" path="/var/lib/kubelet/pods/b35752ab-ccd9-41c1-b954-f02d12a85bc3/volumes"
Jul 01 18:07:40.007203 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:40.007173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:07:40.011073 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:40.011048    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9b9977845-hjnlw"
Jul 01 18:07:45.720859 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:45.720822    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:07:45.721371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:45.721126    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="alertmanager" containerID="cri-o://f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0" gracePeriod=120
Jul 01 18:07:45.721371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:45.721184    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy-metric" containerID="cri-o://fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2" gracePeriod=120
Jul 01 18:07:45.721371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:45.721205    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy-web" containerID="cri-o://2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732" gracePeriod=120
Jul 01 18:07:45.721371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:45.721235    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="config-reloader" containerID="cri-o://ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542" gracePeriod=120
Jul 01 18:07:45.721371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:45.721294    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="prom-label-proxy" containerID="cri-o://c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2" gracePeriod=120
Jul 01 18:07:45.721371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:45.721285    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy" containerID="cri-o://e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93" gracePeriod=120
Jul 01 18:07:46.509887 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.509854    2577 generic.go:356] "Generic (PLEG): container finished" podID="99e99513-6cab-4720-b096-39fd329a09f2" containerID="c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2" exitCode=0
Jul 01 18:07:46.509887 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.509879    2577 generic.go:356] "Generic (PLEG): container finished" podID="99e99513-6cab-4720-b096-39fd329a09f2" containerID="fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2" exitCode=0
Jul 01 18:07:46.509887 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.509885    2577 generic.go:356] "Generic (PLEG): container finished" podID="99e99513-6cab-4720-b096-39fd329a09f2" containerID="e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93" exitCode=0
Jul 01 18:07:46.509887 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.509891    2577 generic.go:356] "Generic (PLEG): container finished" podID="99e99513-6cab-4720-b096-39fd329a09f2" containerID="ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542" exitCode=0
Jul 01 18:07:46.509887 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.509897    2577 generic.go:356] "Generic (PLEG): container finished" podID="99e99513-6cab-4720-b096-39fd329a09f2" containerID="f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0" exitCode=0
Jul 01 18:07:46.510184 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.509921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerDied","Data":"c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2"}
Jul 01 18:07:46.510184 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.509986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerDied","Data":"fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2"}
Jul 01 18:07:46.510184 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.510014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerDied","Data":"e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93"}
Jul 01 18:07:46.510184 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.510039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerDied","Data":"ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542"}
Jul 01 18:07:46.510184 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.510062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerDied","Data":"f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0"}
Jul 01 18:07:46.964940 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:46.964914    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.124415 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124380    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.124639 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124431    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-cluster-tls-config\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.124639 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-tls-assets\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.124639 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-config-out\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.124639 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124602    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.124868 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124673    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.124868 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-main-db\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.124868 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124750    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-main-tls\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.124868 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124782    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-config-volume\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.124868 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124817    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-metrics-client-ca\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.125171 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-trusted-ca-bundle\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.125171 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124905    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-web-config\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.125171 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.124937    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4h5l\" (UniqueName: \"kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-kube-api-access-b4h5l\") pod \"99e99513-6cab-4720-b096-39fd329a09f2\" (UID: \"99e99513-6cab-4720-b096-39fd329a09f2\") "
Jul 01 18:07:47.125756 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.125671    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:47.126860 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.126438    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:47.126860 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.126585    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:07:47.129521 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.129177    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-config-out" (OuterVolumeSpecName: "config-out") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:07:47.129521 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.129488    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:07:47.129872 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.129783    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:47.129872 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.129813    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-kube-api-access-b4h5l" (OuterVolumeSpecName: "kube-api-access-b4h5l") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "kube-api-access-b4h5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:07:47.130010 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.129958    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:47.130070 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.130053    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:47.130165 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.130138    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:47.130205 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.130183    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-config-volume" (OuterVolumeSpecName: "config-volume") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:47.132896 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.132867    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:47.140660 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.140600    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-web-config" (OuterVolumeSpecName: "web-config") pod "99e99513-6cab-4720-b096-39fd329a09f2" (UID: "99e99513-6cab-4720-b096-39fd329a09f2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227102    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-config-out\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227142    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227166    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227183    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-main-db\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227198    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-main-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227213    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-config-volume\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227232    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-metrics-client-ca\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227245    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e99513-6cab-4720-b096-39fd329a09f2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227259    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-web-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227272    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4h5l\" (UniqueName: \"kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-kube-api-access-b4h5l\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227290    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227305    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99e99513-6cab-4720-b096-39fd329a09f2-cluster-tls-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.231394 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.227319    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99e99513-6cab-4720-b096-39fd329a09f2-tls-assets\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:47.517360 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.517272    2577 generic.go:356] "Generic (PLEG): container finished" podID="99e99513-6cab-4720-b096-39fd329a09f2" containerID="2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732" exitCode=0
Jul 01 18:07:47.517360 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.517320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerDied","Data":"2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732"}
Jul 01 18:07:47.517526 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.517363    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.517526 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.517401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e99513-6cab-4720-b096-39fd329a09f2","Type":"ContainerDied","Data":"229ee8551f521ec63fc1d27643a1ae59914426dd3e903ee9fbaa469bde799bc3"}
Jul 01 18:07:47.517526 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.517433    2577 scope.go:117] "RemoveContainer" containerID="c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2"
Jul 01 18:07:47.526321 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.526302    2577 scope.go:117] "RemoveContainer" containerID="fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2"
Jul 01 18:07:47.533507 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.533488    2577 scope.go:117] "RemoveContainer" containerID="e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93"
Jul 01 18:07:47.541128 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.541102    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:07:47.541745 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.541721    2577 scope.go:117] "RemoveContainer" containerID="2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732"
Jul 01 18:07:47.546888 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.546867    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:07:47.549285 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.549262    2577 scope.go:117] "RemoveContainer" containerID="ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542"
Jul 01 18:07:47.556906 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.556886    2577 scope.go:117] "RemoveContainer" containerID="f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0"
Jul 01 18:07:47.564693 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.564673    2577 scope.go:117] "RemoveContainer" containerID="101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92"
Jul 01 18:07:47.571195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571164    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:07:47.571675 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571658    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy-web"
Jul 01 18:07:47.571727 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571679    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy-web"
Jul 01 18:07:47.571727 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571694    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="config-reloader"
Jul 01 18:07:47.571727 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571703    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="config-reloader"
Jul 01 18:07:47.571727 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571714    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy-metric"
Jul 01 18:07:47.571727 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571722    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy-metric"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571734    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" containerName="console"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571742    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" containerName="console"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571758    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571766    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571777    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="prom-label-proxy"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571785    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="prom-label-proxy"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571797    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="init-config-reloader"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571805    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="init-config-reloader"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571825    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b35752ab-ccd9-41c1-b954-f02d12a85bc3" containerName="console"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571832    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b35752ab-ccd9-41c1-b954-f02d12a85bc3" containerName="console"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571841    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="alertmanager"
Jul 01 18:07:47.571890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571849    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="alertmanager"
Jul 01 18:07:47.572231 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571920    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7fdecbf-c6ac-4a1e-84b6-d8e4d6a4c845" containerName="console"
Jul 01 18:07:47.572231 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571933    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b35752ab-ccd9-41c1-b954-f02d12a85bc3" containerName="console"
Jul 01 18:07:47.572231 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571942    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy-web"
Jul 01 18:07:47.572231 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571952    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="config-reloader"
Jul 01 18:07:47.572231 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571959    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy"
Jul 01 18:07:47.572231 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571970    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="prom-label-proxy"
Jul 01 18:07:47.572231 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571978    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="kube-rbac-proxy-metric"
Jul 01 18:07:47.572231 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.571986    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e99513-6cab-4720-b096-39fd329a09f2" containerName="alertmanager"
Jul 01 18:07:47.573568 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.573550    2577 scope.go:117] "RemoveContainer" containerID="c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2"
Jul 01 18:07:47.573948 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:47.573924    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2\": container with ID starting with c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2 not found: ID does not exist" containerID="c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2"
Jul 01 18:07:47.574012 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.573956    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2"} err="failed to get container status \"c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2\": rpc error: code = NotFound desc = could not find container \"c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2\": container with ID starting with c3a48424f69655fc6cf51f7874f702d633036b3ba6a42552665d1f472974e2e2 not found: ID does not exist"
Jul 01 18:07:47.574012 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.573995    2577 scope.go:117] "RemoveContainer" containerID="fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2"
Jul 01 18:07:47.574284 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:47.574259    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2\": container with ID starting with fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2 not found: ID does not exist" containerID="fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2"
Jul 01 18:07:47.574329 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.574294    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2"} err="failed to get container status \"fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2\": rpc error: code = NotFound desc = could not find container \"fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2\": container with ID starting with fa05fc896905973a1bd7a999640dffa8a56481a9bfbf2247f51a26c0ba8761c2 not found: ID does not exist"
Jul 01 18:07:47.574329 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.574318    2577 scope.go:117] "RemoveContainer" containerID="e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93"
Jul 01 18:07:47.574607 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:47.574578    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93\": container with ID starting with e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93 not found: ID does not exist" containerID="e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93"
Jul 01 18:07:47.574729 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.574633    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93"} err="failed to get container status \"e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93\": rpc error: code = NotFound desc = could not find container \"e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93\": container with ID starting with e5ba535ba1c2c3fe07cb70d6d83478204a9eaa70bb7d22629cc636fb8448da93 not found: ID does not exist"
Jul 01 18:07:47.574729 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.574655    2577 scope.go:117] "RemoveContainer" containerID="2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732"
Jul 01 18:07:47.574938 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:47.574919    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732\": container with ID starting with 2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732 not found: ID does not exist" containerID="2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732"
Jul 01 18:07:47.574981 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.574942    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732"} err="failed to get container status \"2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732\": rpc error: code = NotFound desc = could not find container \"2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732\": container with ID starting with 2f6c8ec71b058401e6ed6bec9735ddf44cb62d6c07c6ebbabdc415cbb98f5732 not found: ID does not exist"
Jul 01 18:07:47.574981 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.574957    2577 scope.go:117] "RemoveContainer" containerID="ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542"
Jul 01 18:07:47.575200 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:47.575182    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542\": container with ID starting with ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542 not found: ID does not exist" containerID="ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542"
Jul 01 18:07:47.575236 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.575205    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542"} err="failed to get container status \"ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542\": rpc error: code = NotFound desc = could not find container \"ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542\": container with ID starting with ef14da7471a03e45b770b6e864a8a7c7f0abaa8f8a3191fb5a738e7efb9aa542 not found: ID does not exist"
Jul 01 18:07:47.575236 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.575223    2577 scope.go:117] "RemoveContainer" containerID="f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0"
Jul 01 18:07:47.575476 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:47.575460    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0\": container with ID starting with f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0 not found: ID does not exist" containerID="f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0"
Jul 01 18:07:47.575510 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.575486    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0"} err="failed to get container status \"f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0\": rpc error: code = NotFound desc = could not find container \"f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0\": container with ID starting with f785d11d8c9008923355997fd84e27b440cd39ac44fb8d44ab53fe3c96cbd9b0 not found: ID does not exist"
Jul 01 18:07:47.575510 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.575503    2577 scope.go:117] "RemoveContainer" containerID="101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92"
Jul 01 18:07:47.576102 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:47.576082    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92\": container with ID starting with 101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92 not found: ID does not exist" containerID="101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92"
Jul 01 18:07:47.576175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.576103    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92"} err="failed to get container status \"101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92\": rpc error: code = NotFound desc = could not find container \"101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92\": container with ID starting with 101868cdd0b83deac8612af7cd00bef8591fc4f8e07b67302a4c9155dee05d92 not found: ID does not exist"
Jul 01 18:07:47.576225 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.576200    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.579756 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.579686    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 18:07:47.579881 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.579860    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jdzsf\""
Jul 01 18:07:47.579881 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.579877    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 18:07:47.579998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.579912    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 18:07:47.579998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.579938    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 18:07:47.580084 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.580030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 18:07:47.580136 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.580103    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 18:07:47.580185 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.580150    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 18:07:47.580237 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.580183    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 18:07:47.583979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.583957    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 18:07:47.592847 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.592820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:07:47.731483 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ddea599-b77d-4f0e-9bfc-9603605d1141-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731483 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ddea599-b77d-4f0e-9bfc-9603605d1141-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731755 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8vk\" (UniqueName: \"kubernetes.io/projected/5ddea599-b77d-4f0e-9bfc-9603605d1141-kube-api-access-7r8vk\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731755 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731755 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731755 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731755 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ddea599-b77d-4f0e-9bfc-9603605d1141-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731755 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ddea599-b77d-4f0e-9bfc-9603605d1141-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731963 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ddea599-b77d-4f0e-9bfc-9603605d1141-config-out\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731963 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731963 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731963 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-web-config\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.731963 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.731918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.832585 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.832718 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ddea599-b77d-4f0e-9bfc-9603605d1141-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.832718 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ddea599-b77d-4f0e-9bfc-9603605d1141-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.832718 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8vk\" (UniqueName: \"kubernetes.io/projected/5ddea599-b77d-4f0e-9bfc-9603605d1141-kube-api-access-7r8vk\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.832718 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.832901 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.832901 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.832901 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ddea599-b77d-4f0e-9bfc-9603605d1141-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.832901 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ddea599-b77d-4f0e-9bfc-9603605d1141-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.833086 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ddea599-b77d-4f0e-9bfc-9603605d1141-config-out\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.833086 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.833086 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.833086 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.832994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-web-config\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.834599 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.833961    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ddea599-b77d-4f0e-9bfc-9603605d1141-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.834599 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.834545    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ddea599-b77d-4f0e-9bfc-9603605d1141-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.835680 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.835650    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ddea599-b77d-4f0e-9bfc-9603605d1141-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.836192 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.836070    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.836290 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.836213    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-web-config\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.836290 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.836224    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.836402 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.836362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.836462 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.836404    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ddea599-b77d-4f0e-9bfc-9603605d1141-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.836580 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.836562    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ddea599-b77d-4f0e-9bfc-9603605d1141-config-out\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.836747 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.836725    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.837869 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.837843    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.838542 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.838525    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ddea599-b77d-4f0e-9bfc-9603605d1141-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.841109 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.841091    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8vk\" (UniqueName: \"kubernetes.io/projected/5ddea599-b77d-4f0e-9bfc-9603605d1141-kube-api-access-7r8vk\") pod \"alertmanager-main-0\" (UID: \"5ddea599-b77d-4f0e-9bfc-9603605d1141\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:47.869150 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.869121    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e99513-6cab-4720-b096-39fd329a09f2" path="/var/lib/kubelet/pods/99e99513-6cab-4720-b096-39fd329a09f2/volumes"
Jul 01 18:07:47.887145 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:47.887121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 18:07:48.030439 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:48.030410    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 18:07:48.032177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:48.032151    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddea599_b77d_4f0e_9bfc_9603605d1141.slice/crio-784de0b9905074ee38a8f4b8ae7f4fdbec59f9a812828cc2e32a9e28c4c5d7c7 WatchSource:0}: Error finding container 784de0b9905074ee38a8f4b8ae7f4fdbec59f9a812828cc2e32a9e28c4c5d7c7: Status 404 returned error can't find the container with id 784de0b9905074ee38a8f4b8ae7f4fdbec59f9a812828cc2e32a9e28c4c5d7c7
Jul 01 18:07:48.525141 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:48.525103    2577 generic.go:356] "Generic (PLEG): container finished" podID="5ddea599-b77d-4f0e-9bfc-9603605d1141" containerID="550705aee0f5ac2b2c0dbcde590ed8b1d62a86d0ea9c22a9cb5eaa1e186c80c7" exitCode=0
Jul 01 18:07:48.525141 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:48.525142    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ddea599-b77d-4f0e-9bfc-9603605d1141","Type":"ContainerDied","Data":"550705aee0f5ac2b2c0dbcde590ed8b1d62a86d0ea9c22a9cb5eaa1e186c80c7"}
Jul 01 18:07:48.525399 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:48.525191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ddea599-b77d-4f0e-9bfc-9603605d1141","Type":"ContainerStarted","Data":"784de0b9905074ee38a8f4b8ae7f4fdbec59f9a812828cc2e32a9e28c4c5d7c7"}
Jul 01 18:07:49.533075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.533041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ddea599-b77d-4f0e-9bfc-9603605d1141","Type":"ContainerStarted","Data":"29f5d1753b2b55c99c3b8ffb02d30af77e8950875b624af173b78d4625ab6883"}
Jul 01 18:07:49.533438 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.533113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ddea599-b77d-4f0e-9bfc-9603605d1141","Type":"ContainerStarted","Data":"91728241a4a1accc7f5fe0955f1f9d5393a81541b2cb06108d30647c551e91b7"}
Jul 01 18:07:49.533438 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.533140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ddea599-b77d-4f0e-9bfc-9603605d1141","Type":"ContainerStarted","Data":"b3a1ad5046266079e2f115168b9c9a39259011828b0c9bcb291cd950bd220bbb"}
Jul 01 18:07:49.533438 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.533165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ddea599-b77d-4f0e-9bfc-9603605d1141","Type":"ContainerStarted","Data":"2f12a77ac17937a977d62c11049b3e901288740ce8b145179fbb0dacb65bca06"}
Jul 01 18:07:49.533438 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.533189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ddea599-b77d-4f0e-9bfc-9603605d1141","Type":"ContainerStarted","Data":"cd0298c545d39dbd93eee448b69ba5c958558720efbb5ffbe551c9fcae50e9e7"}
Jul 01 18:07:49.533438 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.533223    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ddea599-b77d-4f0e-9bfc-9603605d1141","Type":"ContainerStarted","Data":"b0ba6ff47ac5dc5669a9843468dd6ca1667f292dc24e1b29e4835c7f19974422"}
Jul 01 18:07:49.562670 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.562586    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.542817249 podStartE2EDuration="2.562541836s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.724587ms" isStatefulPod=false podCreationTimestamp="2026-07-01 18:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:49.56071913 +0000 UTC m=+100.122889768" watchObservedRunningTime="2026-07-01 18:07:49.562541836 +0000 UTC m=+100.124712662"
Jul 01 18:07:49.937402 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.937368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:07:49.937733 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.937694    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="prometheus" containerID="cri-o://3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7" gracePeriod=600
Jul 01 18:07:49.937866 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.937731    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy" containerID="cri-o://533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37" gracePeriod=600
Jul 01 18:07:49.937866 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.937764    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="thanos-sidecar" containerID="cri-o://53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158" gracePeriod=600
Jul 01 18:07:49.937866 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.937786    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy-web" containerID="cri-o://f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2" gracePeriod=600
Jul 01 18:07:49.937866 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.937763    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="config-reloader" containerID="cri-o://526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13" gracePeriod=600
Jul 01 18:07:49.938085 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:49.937773    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60" gracePeriod=600
Jul 01 18:07:50.193572 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.193497    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.258461 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.258431    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-serving-certs-ca-bundle\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.258633 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.258479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-config-out\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.258633 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.258509    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.258633 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.258525    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-metrics-client-certs\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.258633 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.258550    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-kubelet-serving-ca-bundle\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.258633 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.258582    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-config\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.258940 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.258914    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:50.259015 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.258982    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-web-config\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259029    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259137 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259088    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-thanos-prometheus-http-client-file\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259137 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259125    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvgts\" (UniqueName: \"kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-kube-api-access-jvgts\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259229 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259158    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-2\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259278 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259234    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:50.259278 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259235    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-db\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259382 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259323    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-tls-assets\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259382 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259354    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-0\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259476 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259398    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-tls\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259476 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259425    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-1\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259476 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259451    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-grpc-tls\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259734 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259477    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-kube-rbac-proxy\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259734 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259510    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-metrics-client-ca\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259734 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259552    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-trusted-ca-bundle\") pod \"7471ca64-952b-4347-bcfe-7565bef0d03a\" (UID: \"7471ca64-952b-4347-bcfe-7565bef0d03a\") "
Jul 01 18:07:50.259911 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259883    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.259911 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.259905    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.260112 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.260090    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:07:50.260970 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.260184    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:50.260970 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.260693    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:50.261728 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.261474    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:50.261832 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.261810    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:50.262171 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.262099    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:50.263065 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.263037    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:07:50.264916 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.264869    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:50.265012 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.264982    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:50.265165 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.265069    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-config-out" (OuterVolumeSpecName: "config-out") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:07:50.265258 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.265192    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:50.265258 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.265245    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:50.265397 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.265357    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-config" (OuterVolumeSpecName: "config") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:50.265590 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.265564    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:50.265937 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.265914    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:50.266015 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.265946    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:07:50.266284 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.266264    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-kube-api-access-jvgts" (OuterVolumeSpecName: "kube-api-access-jvgts") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "kube-api-access-jvgts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:07:50.276210 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.276183    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-web-config" (OuterVolumeSpecName: "web-config") pod "7471ca64-952b-4347-bcfe-7565bef0d03a" (UID: "7471ca64-952b-4347-bcfe-7565bef0d03a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:07:50.360784 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360752    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvgts\" (UniqueName: \"kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-kube-api-access-jvgts\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.360784 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360777    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.360784 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360788    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-db\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360797    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7471ca64-952b-4347-bcfe-7565bef0d03a-tls-assets\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360808    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360816    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360824    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360832    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-grpc-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360839    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-kube-rbac-proxy\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360847    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-configmap-metrics-client-ca\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360856    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7471ca64-952b-4347-bcfe-7565bef0d03a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360864    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7471ca64-952b-4347-bcfe-7565bef0d03a-config-out\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360872    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360881    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-metrics-client-certs\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360889    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360898    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-web-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360907    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.361002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.360915    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7471ca64-952b-4347-bcfe-7565bef0d03a-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:07:50.541927 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.541854    2577 generic.go:356] "Generic (PLEG): container finished" podID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerID="c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60" exitCode=0
Jul 01 18:07:50.541927 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.541877    2577 generic.go:356] "Generic (PLEG): container finished" podID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerID="533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37" exitCode=0
Jul 01 18:07:50.541927 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.541883    2577 generic.go:356] "Generic (PLEG): container finished" podID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerID="f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2" exitCode=0
Jul 01 18:07:50.541927 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.541889    2577 generic.go:356] "Generic (PLEG): container finished" podID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerID="53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158" exitCode=0
Jul 01 18:07:50.541927 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.541893    2577 generic.go:356] "Generic (PLEG): container finished" podID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerID="526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13" exitCode=0
Jul 01 18:07:50.541927 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.541898    2577 generic.go:356] "Generic (PLEG): container finished" podID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerID="3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7" exitCode=0
Jul 01 18:07:50.542504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.541940    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.542504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.541939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerDied","Data":"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"}
Jul 01 18:07:50.542504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.542019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerDied","Data":"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"}
Jul 01 18:07:50.542504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.542040    2577 scope.go:117] "RemoveContainer" containerID="c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"
Jul 01 18:07:50.542504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.542063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerDied","Data":"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"}
Jul 01 18:07:50.542504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.542107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerDied","Data":"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"}
Jul 01 18:07:50.542504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.542149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerDied","Data":"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"}
Jul 01 18:07:50.542504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.542191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerDied","Data":"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"}
Jul 01 18:07:50.542504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.542233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7471ca64-952b-4347-bcfe-7565bef0d03a","Type":"ContainerDied","Data":"108a139cf5f63574d8e9a21b0e876ad8218c00151e28125328cc1d561d482da6"}
Jul 01 18:07:50.550005 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.549987    2577 scope.go:117] "RemoveContainer" containerID="533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"
Jul 01 18:07:50.562480 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.562461    2577 scope.go:117] "RemoveContainer" containerID="f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"
Jul 01 18:07:50.568815 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.568789    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:07:50.571798 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.571772    2577 scope.go:117] "RemoveContainer" containerID="53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"
Jul 01 18:07:50.573390 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.573370    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:07:50.579035 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.579017    2577 scope.go:117] "RemoveContainer" containerID="526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"
Jul 01 18:07:50.588711 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.588646    2577 scope.go:117] "RemoveContainer" containerID="3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"
Jul 01 18:07:50.596516 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.596498    2577 scope.go:117] "RemoveContainer" containerID="4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"
Jul 01 18:07:50.599091 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599068    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:07:50.599399 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599387    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="thanos-sidecar"
Jul 01 18:07:50.599452 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599401    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="thanos-sidecar"
Jul 01 18:07:50.599452 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599414    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="init-config-reloader"
Jul 01 18:07:50.599452 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599420    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="init-config-reloader"
Jul 01 18:07:50.599452 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599434    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy-web"
Jul 01 18:07:50.599452 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599439    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy-web"
Jul 01 18:07:50.599452 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599449    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="prometheus"
Jul 01 18:07:50.599452 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599455    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="prometheus"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599461    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599478    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599489    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="config-reloader"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599495    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="config-reloader"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599501    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy-thanos"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599506    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy-thanos"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599553    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="prometheus"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599560    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599566    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy-web"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599572    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="thanos-sidecar"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599578    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="config-reloader"
Jul 01 18:07:50.599667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.599584    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" containerName="kube-rbac-proxy-thanos"
Jul 01 18:07:50.604466 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.604445    2577 scope.go:117] "RemoveContainer" containerID="c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"
Jul 01 18:07:50.604767 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:50.604747    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": container with ID starting with c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60 not found: ID does not exist" containerID="c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"
Jul 01 18:07:50.604810 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.604777    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"} err="failed to get container status \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": rpc error: code = NotFound desc = could not find container \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": container with ID starting with c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60 not found: ID does not exist"
Jul 01 18:07:50.604810 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.604797    2577 scope.go:117] "RemoveContainer" containerID="533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"
Jul 01 18:07:50.605013 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:50.604998    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": container with ID starting with 533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37 not found: ID does not exist" containerID="533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"
Jul 01 18:07:50.605046 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.605017    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"} err="failed to get container status \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": rpc error: code = NotFound desc = could not find container \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": container with ID starting with 533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37 not found: ID does not exist"
Jul 01 18:07:50.605046 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.605028    2577 scope.go:117] "RemoveContainer" containerID="f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"
Jul 01 18:07:50.605253 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:50.605235    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": container with ID starting with f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2 not found: ID does not exist" containerID="f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"
Jul 01 18:07:50.605296 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.605261    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"} err="failed to get container status \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": rpc error: code = NotFound desc = could not find container \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": container with ID starting with f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2 not found: ID does not exist"
Jul 01 18:07:50.605296 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.605277    2577 scope.go:117] "RemoveContainer" containerID="53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"
Jul 01 18:07:50.605502 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:50.605488    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": container with ID starting with 53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158 not found: ID does not exist" containerID="53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"
Jul 01 18:07:50.605535 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.605511    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"} err="failed to get container status \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": rpc error: code = NotFound desc = could not find container \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": container with ID starting with 53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158 not found: ID does not exist"
Jul 01 18:07:50.605535 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.605525    2577 scope.go:117] "RemoveContainer" containerID="526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"
Jul 01 18:07:50.605788 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:50.605735    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": container with ID starting with 526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13 not found: ID does not exist" containerID="526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"
Jul 01 18:07:50.605788 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.605760    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"} err="failed to get container status \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": rpc error: code = NotFound desc = could not find container \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": container with ID starting with 526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13 not found: ID does not exist"
Jul 01 18:07:50.605788 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.605780    2577 scope.go:117] "RemoveContainer" containerID="3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"
Jul 01 18:07:50.606136 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:50.606117    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": container with ID starting with 3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7 not found: ID does not exist" containerID="3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"
Jul 01 18:07:50.606232 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.606142    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"} err="failed to get container status \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": rpc error: code = NotFound desc = could not find container \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": container with ID starting with 3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7 not found: ID does not exist"
Jul 01 18:07:50.606232 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.606162    2577 scope.go:117] "RemoveContainer" containerID="4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"
Jul 01 18:07:50.606481 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:07:50.606460    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": container with ID starting with 4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66 not found: ID does not exist" containerID="4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"
Jul 01 18:07:50.606537 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.606484    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"} err="failed to get container status \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": rpc error: code = NotFound desc = could not find container \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": container with ID starting with 4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66 not found: ID does not exist"
Jul 01 18:07:50.606537 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.606504    2577 scope.go:117] "RemoveContainer" containerID="c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"
Jul 01 18:07:50.606750 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.606730    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"} err="failed to get container status \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": rpc error: code = NotFound desc = could not find container \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": container with ID starting with c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60 not found: ID does not exist"
Jul 01 18:07:50.606832 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.606751    2577 scope.go:117] "RemoveContainer" containerID="533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"
Jul 01 18:07:50.607074 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.607055    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"} err="failed to get container status \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": rpc error: code = NotFound desc = could not find container \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": container with ID starting with 533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37 not found: ID does not exist"
Jul 01 18:07:50.607153 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.607075    2577 scope.go:117] "RemoveContainer" containerID="f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"
Jul 01 18:07:50.607327 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.607309    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"} err="failed to get container status \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": rpc error: code = NotFound desc = could not find container \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": container with ID starting with f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2 not found: ID does not exist"
Jul 01 18:07:50.607393 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.607328    2577 scope.go:117] "RemoveContainer" containerID="53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"
Jul 01 18:07:50.607578 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.607556    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"} err="failed to get container status \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": rpc error: code = NotFound desc = could not find container \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": container with ID starting with 53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158 not found: ID does not exist"
Jul 01 18:07:50.607646 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.607581    2577 scope.go:117] "RemoveContainer" containerID="526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"
Jul 01 18:07:50.607872 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.607854    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"} err="failed to get container status \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": rpc error: code = NotFound desc = could not find container \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": container with ID starting with 526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13 not found: ID does not exist"
Jul 01 18:07:50.607912 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.607874    2577 scope.go:117] "RemoveContainer" containerID="3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"
Jul 01 18:07:50.608087 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.608070    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"} err="failed to get container status \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": rpc error: code = NotFound desc = could not find container \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": container with ID starting with 3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7 not found: ID does not exist"
Jul 01 18:07:50.608159 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.608089    2577 scope.go:117] "RemoveContainer" containerID="4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"
Jul 01 18:07:50.608306 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.608288    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"} err="failed to get container status \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": rpc error: code = NotFound desc = could not find container \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": container with ID starting with 4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66 not found: ID does not exist"
Jul 01 18:07:50.608357 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.608307    2577 scope.go:117] "RemoveContainer" containerID="c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"
Jul 01 18:07:50.608501 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.608485    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"} err="failed to get container status \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": rpc error: code = NotFound desc = could not find container \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": container with ID starting with c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60 not found: ID does not exist"
Jul 01 18:07:50.608544 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.608510    2577 scope.go:117] "RemoveContainer" containerID="533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"
Jul 01 18:07:50.608733 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.608716    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"} err="failed to get container status \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": rpc error: code = NotFound desc = could not find container \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": container with ID starting with 533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37 not found: ID does not exist"
Jul 01 18:07:50.608853 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.608735    2577 scope.go:117] "RemoveContainer" containerID="f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"
Jul 01 18:07:50.608938 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.608923    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"} err="failed to get container status \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": rpc error: code = NotFound desc = could not find container \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": container with ID starting with f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2 not found: ID does not exist"
Jul 01 18:07:50.608985 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.608939    2577 scope.go:117] "RemoveContainer" containerID="53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"
Jul 01 18:07:50.609102 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.609085    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"} err="failed to get container status \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": rpc error: code = NotFound desc = could not find container \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": container with ID starting with 53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158 not found: ID does not exist"
Jul 01 18:07:50.609144 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.609103    2577 scope.go:117] "RemoveContainer" containerID="526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"
Jul 01 18:07:50.609289 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.609273    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"} err="failed to get container status \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": rpc error: code = NotFound desc = could not find container \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": container with ID starting with 526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13 not found: ID does not exist"
Jul 01 18:07:50.609289 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.609288    2577 scope.go:117] "RemoveContainer" containerID="3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"
Jul 01 18:07:50.609496 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.609480    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"} err="failed to get container status \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": rpc error: code = NotFound desc = could not find container \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": container with ID starting with 3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7 not found: ID does not exist"
Jul 01 18:07:50.609534 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.609497    2577 scope.go:117] "RemoveContainer" containerID="4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"
Jul 01 18:07:50.609755 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.609736    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"} err="failed to get container status \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": rpc error: code = NotFound desc = could not find container \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": container with ID starting with 4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66 not found: ID does not exist"
Jul 01 18:07:50.609755 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.609755    2577 scope.go:117] "RemoveContainer" containerID="c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"
Jul 01 18:07:50.610007 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.609986    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"} err="failed to get container status \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": rpc error: code = NotFound desc = could not find container \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": container with ID starting with c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60 not found: ID does not exist"
Jul 01 18:07:50.610059 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.610011    2577 scope.go:117] "RemoveContainer" containerID="533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"
Jul 01 18:07:50.610258 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.610238    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"} err="failed to get container status \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": rpc error: code = NotFound desc = could not find container \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": container with ID starting with 533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37 not found: ID does not exist"
Jul 01 18:07:50.610330 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.610259    2577 scope.go:117] "RemoveContainer" containerID="f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"
Jul 01 18:07:50.610482 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.610459    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"} err="failed to get container status \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": rpc error: code = NotFound desc = could not find container \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": container with ID starting with f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2 not found: ID does not exist"
Jul 01 18:07:50.610482 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.610482    2577 scope.go:117] "RemoveContainer" containerID="53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"
Jul 01 18:07:50.610754 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.610736    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"} err="failed to get container status \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": rpc error: code = NotFound desc = could not find container \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": container with ID starting with 53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158 not found: ID does not exist"
Jul 01 18:07:50.610754 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.610754    2577 scope.go:117] "RemoveContainer" containerID="526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"
Jul 01 18:07:50.611013 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.610997    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"} err="failed to get container status \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": rpc error: code = NotFound desc = could not find container \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": container with ID starting with 526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13 not found: ID does not exist"
Jul 01 18:07:50.611064 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.611015    2577 scope.go:117] "RemoveContainer" containerID="3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"
Jul 01 18:07:50.611231 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.611213    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"} err="failed to get container status \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": rpc error: code = NotFound desc = could not find container \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": container with ID starting with 3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7 not found: ID does not exist"
Jul 01 18:07:50.611277 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.611231    2577 scope.go:117] "RemoveContainer" containerID="4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"
Jul 01 18:07:50.611453 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.611435    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"} err="failed to get container status \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": rpc error: code = NotFound desc = could not find container \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": container with ID starting with 4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66 not found: ID does not exist"
Jul 01 18:07:50.611495 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.611453    2577 scope.go:117] "RemoveContainer" containerID="c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"
Jul 01 18:07:50.611707 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.611686    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"} err="failed to get container status \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": rpc error: code = NotFound desc = could not find container \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": container with ID starting with c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60 not found: ID does not exist"
Jul 01 18:07:50.611707 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.611706    2577 scope.go:117] "RemoveContainer" containerID="533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"
Jul 01 18:07:50.611934 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.611915    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"} err="failed to get container status \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": rpc error: code = NotFound desc = could not find container \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": container with ID starting with 533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37 not found: ID does not exist"
Jul 01 18:07:50.611989 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.611935    2577 scope.go:117] "RemoveContainer" containerID="f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"
Jul 01 18:07:50.612171 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.612153    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"} err="failed to get container status \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": rpc error: code = NotFound desc = could not find container \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": container with ID starting with f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2 not found: ID does not exist"
Jul 01 18:07:50.612247 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.612173    2577 scope.go:117] "RemoveContainer" containerID="53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"
Jul 01 18:07:50.612407 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.612391    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"} err="failed to get container status \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": rpc error: code = NotFound desc = could not find container \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": container with ID starting with 53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158 not found: ID does not exist"
Jul 01 18:07:50.612456 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.612407    2577 scope.go:117] "RemoveContainer" containerID="526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"
Jul 01 18:07:50.612602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.612587    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"} err="failed to get container status \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": rpc error: code = NotFound desc = could not find container \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": container with ID starting with 526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13 not found: ID does not exist"
Jul 01 18:07:50.612670 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.612603    2577 scope.go:117] "RemoveContainer" containerID="3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"
Jul 01 18:07:50.612876 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.612858    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"} err="failed to get container status \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": rpc error: code = NotFound desc = could not find container \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": container with ID starting with 3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7 not found: ID does not exist"
Jul 01 18:07:50.612945 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.612877    2577 scope.go:117] "RemoveContainer" containerID="4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"
Jul 01 18:07:50.613096 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.613080    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"} err="failed to get container status \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": rpc error: code = NotFound desc = could not find container \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": container with ID starting with 4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66 not found: ID does not exist"
Jul 01 18:07:50.613143 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.613097    2577 scope.go:117] "RemoveContainer" containerID="c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"
Jul 01 18:07:50.613273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.613258    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60"} err="failed to get container status \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": rpc error: code = NotFound desc = could not find container \"c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60\": container with ID starting with c7e6f63568196d0255a9f5397004a101630a91f4dbcde3ac4efaa44bb9e6ef60 not found: ID does not exist"
Jul 01 18:07:50.613322 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.613276    2577 scope.go:117] "RemoveContainer" containerID="533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"
Jul 01 18:07:50.613494 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.613476    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37"} err="failed to get container status \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": rpc error: code = NotFound desc = could not find container \"533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37\": container with ID starting with 533cca893725cbf30012977e5108e46d8730cd351bb7dad59682e2fce46aef37 not found: ID does not exist"
Jul 01 18:07:50.613532 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.613494    2577 scope.go:117] "RemoveContainer" containerID="f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"
Jul 01 18:07:50.613742 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.613724    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2"} err="failed to get container status \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": rpc error: code = NotFound desc = could not find container \"f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2\": container with ID starting with f5fe899a94c083314f5619492a693b0ba8d9a565ec19b43d8cfff2c51df237e2 not found: ID does not exist"
Jul 01 18:07:50.613848 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.613741    2577 scope.go:117] "RemoveContainer" containerID="53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"
Jul 01 18:07:50.613957 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.613939    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158"} err="failed to get container status \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": rpc error: code = NotFound desc = could not find container \"53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158\": container with ID starting with 53811b8ec14455c419a78c0a0486a02ad1e6da4139dd13aef7fc7d49ebdb4158 not found: ID does not exist"
Jul 01 18:07:50.614000 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.613966    2577 scope.go:117] "RemoveContainer" containerID="526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"
Jul 01 18:07:50.614186 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.614155    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13"} err="failed to get container status \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": rpc error: code = NotFound desc = could not find container \"526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13\": container with ID starting with 526516c3faae8e460cdf4e6b9ef926ce0b8b7e5d4af59611327f4a304faf7f13 not found: ID does not exist"
Jul 01 18:07:50.614186 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.614178    2577 scope.go:117] "RemoveContainer" containerID="3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"
Jul 01 18:07:50.614409 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.614384    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7"} err="failed to get container status \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": rpc error: code = NotFound desc = could not find container \"3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7\": container with ID starting with 3568d88be1166abc32a939972f305213406c72d4fc0976d018fcb65db376e4b7 not found: ID does not exist"
Jul 01 18:07:50.614460 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.614408    2577 scope.go:117] "RemoveContainer" containerID="4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"
Jul 01 18:07:50.614640 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.614596    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66"} err="failed to get container status \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": rpc error: code = NotFound desc = could not find container \"4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66\": container with ID starting with 4bbb5ae70d92a1c52ec9cf256f089a8e52d9cf4f64d9b769a3c27fc9921a1c66 not found: ID does not exist"
Jul 01 18:07:50.620898 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.620430    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:07:50.620898 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.620553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.623906 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.623885    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 18:07:50.624044 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.624023    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 18:07:50.624636 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.624587    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 18:07:50.624844 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.624821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 18:07:50.624960 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.624889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 18:07:50.625056 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.624968    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wxjd5\""
Jul 01 18:07:50.625132 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.625093    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 18:07:50.625132 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.625117    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 18:07:50.625300 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.625166    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 18:07:50.625300 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.625181    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 18:07:50.625417 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.625387    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6fjrtgm03bb4v\""
Jul 01 18:07:50.625417 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.625400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 18:07:50.625508 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.625392    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 18:07:50.625744 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.625729    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 18:07:50.627947 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.627922    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 18:07:50.630395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.630376    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 18:07:50.663507 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663670 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663670 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663670 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663670 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2rwk\" (UniqueName: \"kubernetes.io/projected/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-kube-api-access-b2rwk\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663843 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663843 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663843 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663843 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-config\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.663979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.663975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.664233 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.664006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.664233 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.664029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.664233 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.664044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.764931 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.764890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.764931 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.764931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.764955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.764983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2rwk\" (UniqueName: \"kubernetes.io/projected/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-kube-api-access-b2rwk\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-config\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.765589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.766057    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.766695    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.767983    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.768235    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.768263    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.768328    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.768876    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.768883    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.769284    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.769489    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.769487    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.769978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.769639    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.770388 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.770043    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.770388 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.770068    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.770388 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.770110    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.770866 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.770845    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.770950 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.770903    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.771047 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.771029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.771380 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.771362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-config\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.773560 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.773538    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2rwk\" (UniqueName: \"kubernetes.io/projected/9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094-kube-api-access-b2rwk\") pod \"prometheus-k8s-0\" (UID: \"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:50.935099 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:50.935058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:07:51.089428 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:51.089395    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:07:51.092451 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:07:51.092421    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3cb9ec_a6b0_4f0f_a4eb_9064a37f3094.slice/crio-20aaf70646cc55b4f9550163fb0e88d4fa6a6d34d7a565b9e639d4e95119873a WatchSource:0}: Error finding container 20aaf70646cc55b4f9550163fb0e88d4fa6a6d34d7a565b9e639d4e95119873a: Status 404 returned error can't find the container with id 20aaf70646cc55b4f9550163fb0e88d4fa6a6d34d7a565b9e639d4e95119873a
Jul 01 18:07:51.548631 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:51.548537    2577 generic.go:356] "Generic (PLEG): container finished" podID="9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094" containerID="355e3332f2ac52a2ced0e96f7508aca368808755589251c793d090a6113742a0" exitCode=0
Jul 01 18:07:51.549073 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:51.548646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094","Type":"ContainerDied","Data":"355e3332f2ac52a2ced0e96f7508aca368808755589251c793d090a6113742a0"}
Jul 01 18:07:51.549073 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:51.548716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094","Type":"ContainerStarted","Data":"20aaf70646cc55b4f9550163fb0e88d4fa6a6d34d7a565b9e639d4e95119873a"}
Jul 01 18:07:51.870490 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:51.870464    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7471ca64-952b-4347-bcfe-7565bef0d03a" path="/var/lib/kubelet/pods/7471ca64-952b-4347-bcfe-7565bef0d03a/volumes"
Jul 01 18:07:52.558590 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:52.558551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094","Type":"ContainerStarted","Data":"d9c9e146f07ef686425ba9dc1174cbfad2fb11d1a190ce6d09161543f857af5b"}
Jul 01 18:07:52.559053 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:52.558642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094","Type":"ContainerStarted","Data":"77fc6cfb4e27d6ddeb0868cebea8fcba980fb1a9705c96706df0c941cd6e64f1"}
Jul 01 18:07:52.559053 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:52.558672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094","Type":"ContainerStarted","Data":"2e422484082e61973fd16352fc024f93f56629707a57a678e9f6d1bb92c0c5ab"}
Jul 01 18:07:52.559053 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:52.558698    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094","Type":"ContainerStarted","Data":"71f0dd0c60f0538fa65c9478f1cb42c9c1bac8fea490e5bd4e384812561d58f0"}
Jul 01 18:07:52.559053 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:52.558720    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094","Type":"ContainerStarted","Data":"e561b723d67544f608df8d4a9316a7523603f7bf08342186d5786ba8b353dbe0"}
Jul 01 18:07:52.559053 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:52.558743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094","Type":"ContainerStarted","Data":"8590281dfdcf5ede4a2a8034246ec4f989e93142f54df7cf6e472e539a39e021"}
Jul 01 18:07:52.589885 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:52.589828    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.565402869 podStartE2EDuration="2.589811835s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.408966ms" isStatefulPod=false podCreationTimestamp="2026-07-01 18:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:07:52.58735314 +0000 UTC m=+103.149523789" watchObservedRunningTime="2026-07-01 18:07:52.589811835 +0000 UTC m=+103.151982471"
Jul 01 18:07:53.002246 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:53.002208    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cf6b987cc-kl2tp"]
Jul 01 18:07:53.393693 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:53.393668    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-stnmw"
Jul 01 18:07:55.936215 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:07:55.936182    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:08:18.022718 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.022653    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5cf6b987cc-kl2tp" podUID="13481e8d-918a-4482-b6a5-21d73329b2f6" containerName="console" containerID="cri-o://a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466" gracePeriod=15
Jul 01 18:08:18.263738 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.263711    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:08:18.416129 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416096    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-oauth-serving-cert\") pod \"13481e8d-918a-4482-b6a5-21d73329b2f6\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") "
Jul 01 18:08:18.416302 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-service-ca\") pod \"13481e8d-918a-4482-b6a5-21d73329b2f6\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") "
Jul 01 18:08:18.416302 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct8pz\" (UniqueName: \"kubernetes.io/projected/13481e8d-918a-4482-b6a5-21d73329b2f6-kube-api-access-ct8pz\") pod \"13481e8d-918a-4482-b6a5-21d73329b2f6\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") "
Jul 01 18:08:18.416419 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-serving-cert\") pod \"13481e8d-918a-4482-b6a5-21d73329b2f6\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") "
Jul 01 18:08:18.416419 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416348    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-console-config\") pod \"13481e8d-918a-4482-b6a5-21d73329b2f6\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") "
Jul 01 18:08:18.416419 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416387    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-oauth-config\") pod \"13481e8d-918a-4482-b6a5-21d73329b2f6\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") "
Jul 01 18:08:18.416658 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416434    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-trusted-ca-bundle\") pod \"13481e8d-918a-4482-b6a5-21d73329b2f6\" (UID: \"13481e8d-918a-4482-b6a5-21d73329b2f6\") "
Jul 01 18:08:18.416658 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416557    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "13481e8d-918a-4482-b6a5-21d73329b2f6" (UID: "13481e8d-918a-4482-b6a5-21d73329b2f6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:08:18.416766 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416722    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-service-ca" (OuterVolumeSpecName: "service-ca") pod "13481e8d-918a-4482-b6a5-21d73329b2f6" (UID: "13481e8d-918a-4482-b6a5-21d73329b2f6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:08:18.416820 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416783    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-console-config" (OuterVolumeSpecName: "console-config") pod "13481e8d-918a-4482-b6a5-21d73329b2f6" (UID: "13481e8d-918a-4482-b6a5-21d73329b2f6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:08:18.416868 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416830    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-oauth-serving-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:08:18.416868 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.416853    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-service-ca\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:08:18.417027 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.417003    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "13481e8d-918a-4482-b6a5-21d73329b2f6" (UID: "13481e8d-918a-4482-b6a5-21d73329b2f6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:08:18.418648 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.418602    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13481e8d-918a-4482-b6a5-21d73329b2f6-kube-api-access-ct8pz" (OuterVolumeSpecName: "kube-api-access-ct8pz") pod "13481e8d-918a-4482-b6a5-21d73329b2f6" (UID: "13481e8d-918a-4482-b6a5-21d73329b2f6"). InnerVolumeSpecName "kube-api-access-ct8pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:08:18.418767 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.418724    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "13481e8d-918a-4482-b6a5-21d73329b2f6" (UID: "13481e8d-918a-4482-b6a5-21d73329b2f6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:08:18.418808 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.418769    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "13481e8d-918a-4482-b6a5-21d73329b2f6" (UID: "13481e8d-918a-4482-b6a5-21d73329b2f6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:08:18.518208 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.518164    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-trusted-ca-bundle\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:08:18.518208 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.518201    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct8pz\" (UniqueName: \"kubernetes.io/projected/13481e8d-918a-4482-b6a5-21d73329b2f6-kube-api-access-ct8pz\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:08:18.518208 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.518211    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-serving-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:08:18.518456 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.518221    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13481e8d-918a-4482-b6a5-21d73329b2f6-console-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:08:18.518456 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.518231    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13481e8d-918a-4482-b6a5-21d73329b2f6-console-oauth-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:08:18.676691 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.676591    2577 generic.go:356] "Generic (PLEG): container finished" podID="13481e8d-918a-4482-b6a5-21d73329b2f6" containerID="a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466" exitCode=0
Jul 01 18:08:18.676691 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.676663    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf6b987cc-kl2tp"
Jul 01 18:08:18.676898 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.676666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf6b987cc-kl2tp" event={"ID":"13481e8d-918a-4482-b6a5-21d73329b2f6","Type":"ContainerDied","Data":"a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466"}
Jul 01 18:08:18.676898 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.676780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf6b987cc-kl2tp" event={"ID":"13481e8d-918a-4482-b6a5-21d73329b2f6","Type":"ContainerDied","Data":"fb08c7832c3a67e94293e6fef7e16819757785fe946801c438c3d767534b479d"}
Jul 01 18:08:18.676898 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.676800    2577 scope.go:117] "RemoveContainer" containerID="a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466"
Jul 01 18:08:18.685564 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.685545    2577 scope.go:117] "RemoveContainer" containerID="a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466"
Jul 01 18:08:18.685902 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:08:18.685879    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466\": container with ID starting with a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466 not found: ID does not exist" containerID="a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466"
Jul 01 18:08:18.685984 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.685912    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466"} err="failed to get container status \"a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466\": rpc error: code = NotFound desc = could not find container \"a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466\": container with ID starting with a5b226dfdd5de3bca0733f5336b48f8ebb975df1336796ed5dc5865f14f18466 not found: ID does not exist"
Jul 01 18:08:18.697151 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.697118    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cf6b987cc-kl2tp"]
Jul 01 18:08:18.700890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:18.700867    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cf6b987cc-kl2tp"]
Jul 01 18:08:19.869296 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:19.869258    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13481e8d-918a-4482-b6a5-21d73329b2f6" path="/var/lib/kubelet/pods/13481e8d-918a-4482-b6a5-21d73329b2f6/volumes"
Jul 01 18:08:50.935872 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:50.935828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:08:50.952081 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:50.952051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:08:51.831656 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:08:51.831602    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:09:13.458651 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.458605    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-4wl7c"]
Jul 01 18:09:13.459004 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.458958    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13481e8d-918a-4482-b6a5-21d73329b2f6" containerName="console"
Jul 01 18:09:13.459004 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.458970    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="13481e8d-918a-4482-b6a5-21d73329b2f6" containerName="console"
Jul 01 18:09:13.459073 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.459031    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="13481e8d-918a-4482-b6a5-21d73329b2f6" containerName="console"
Jul 01 18:09:13.461540 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.461521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:13.464297 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.464267    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 01 18:09:13.464297 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.464281    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 01 18:09:13.464297 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.464272    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 01 18:09:13.464526 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.464331    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 01 18:09:13.465080 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.465061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-jwtwh\""
Jul 01 18:09:13.465213 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.465109    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 01 18:09:13.473592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.473566    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-4wl7c"]
Jul 01 18:09:13.607779 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.607737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7s6x\" (UniqueName: \"kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-kube-api-access-n7s6x\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:13.607967 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.607812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b034c950-2b32-41a4-9714-d33cff34aeca-cabundle0\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:13.607967 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.607833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-certificates\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:13.709012 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.708913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b034c950-2b32-41a4-9714-d33cff34aeca-cabundle0\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:13.709012 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.708951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-certificates\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:13.709012 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.708997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7s6x\" (UniqueName: \"kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-kube-api-access-n7s6x\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:13.709292 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:09:13.709081    2577 secret.go:281] references non-existent secret key: ca.crt
Jul 01 18:09:13.709292 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:09:13.709101    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 01 18:09:13.709292 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:09:13.709110    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-4wl7c: references non-existent secret key: ca.crt
Jul 01 18:09:13.709292 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:09:13.709173    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-certificates podName:b034c950-2b32-41a4-9714-d33cff34aeca nodeName:}" failed. No retries permitted until 2026-07-01 18:09:14.209154829 +0000 UTC m=+184.771325446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-certificates") pod "keda-operator-68bff59c-4wl7c" (UID: "b034c950-2b32-41a4-9714-d33cff34aeca") : references non-existent secret key: ca.crt
Jul 01 18:09:13.709648 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.709628    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b034c950-2b32-41a4-9714-d33cff34aeca-cabundle0\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:13.717800 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:13.717775    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7s6x\" (UniqueName: \"kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-kube-api-access-n7s6x\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:14.214442 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:14.214398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-certificates\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:14.214649 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:09:14.214555    2577 secret.go:281] references non-existent secret key: ca.crt
Jul 01 18:09:14.214649 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:09:14.214572    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 01 18:09:14.214649 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:09:14.214580    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-4wl7c: references non-existent secret key: ca.crt
Jul 01 18:09:14.214771 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:09:14.214658    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-certificates podName:b034c950-2b32-41a4-9714-d33cff34aeca nodeName:}" failed. No retries permitted until 2026-07-01 18:09:15.214642758 +0000 UTC m=+185.776813377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-certificates") pod "keda-operator-68bff59c-4wl7c" (UID: "b034c950-2b32-41a4-9714-d33cff34aeca") : references non-existent secret key: ca.crt
Jul 01 18:09:15.224765 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:15.224722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-certificates\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:15.227430 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:15.227405    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b034c950-2b32-41a4-9714-d33cff34aeca-certificates\") pod \"keda-operator-68bff59c-4wl7c\" (UID: \"b034c950-2b32-41a4-9714-d33cff34aeca\") " pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:15.272639 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:15.272591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:15.416764 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:15.416729    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-4wl7c"]
Jul 01 18:09:15.420526 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:09:15.420496    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb034c950_2b32_41a4_9714_d33cff34aeca.slice/crio-8398812d8649bee5b22e74b3a1d307d6d51acde9a5b1a71760cd8cdd52f22eeb WatchSource:0}: Error finding container 8398812d8649bee5b22e74b3a1d307d6d51acde9a5b1a71760cd8cdd52f22eeb: Status 404 returned error can't find the container with id 8398812d8649bee5b22e74b3a1d307d6d51acde9a5b1a71760cd8cdd52f22eeb
Jul 01 18:09:15.918026 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:15.917986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-4wl7c" event={"ID":"b034c950-2b32-41a4-9714-d33cff34aeca","Type":"ContainerStarted","Data":"8398812d8649bee5b22e74b3a1d307d6d51acde9a5b1a71760cd8cdd52f22eeb"}
Jul 01 18:09:19.937269 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:19.937175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-4wl7c" event={"ID":"b034c950-2b32-41a4-9714-d33cff34aeca","Type":"ContainerStarted","Data":"4d8d06e9d3bec0feacc795a1e45a74c782294d337284630b2f902ca92fb4f1a8"}
Jul 01 18:09:19.937729 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:19.937287    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:09:19.954404 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:19.954354    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-4wl7c" podStartSLOduration=2.738703065 podStartE2EDuration="6.954337883s" totalImagesPullingTime="4.215634818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:09:19.952977681 +0000 UTC m=+190.515148318" watchObservedRunningTime="2026-07-01 18:09:19.954337883 +0000 UTC m=+190.516508519"
Jul 01 18:09:40.943910 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:09:40.943882    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-4wl7c"
Jul 01 18:11:09.802574 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:09.802547    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:11:09.806762 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:09.806733    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:11:09.808761 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:09.808742    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 18:11:27.749435 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:27.749400    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-jwws6"]
Jul 01 18:11:27.752759 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:27.752734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-jwws6"
Jul 01 18:11:27.756514 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:27.756478    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 01 18:11:27.756718 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:27.756699    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-8cb82\""
Jul 01 18:11:27.756811 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:27.756751    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 18:11:27.756811 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:27.756760    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 18:11:27.763341 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:27.763315    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-jwws6"]
Jul 01 18:11:27.903250 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:27.903215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkgn2\" (UniqueName: \"kubernetes.io/projected/22235a60-9ad3-4ad5-8a29-2377aad43187-kube-api-access-hkgn2\") pod \"model-serving-api-fd65d7d6-jwws6\" (UID: \"22235a60-9ad3-4ad5-8a29-2377aad43187\") " pod="kserve/model-serving-api-fd65d7d6-jwws6"
Jul 01 18:11:27.903250 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:27.903255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22235a60-9ad3-4ad5-8a29-2377aad43187-tls-certs\") pod \"model-serving-api-fd65d7d6-jwws6\" (UID: \"22235a60-9ad3-4ad5-8a29-2377aad43187\") " pod="kserve/model-serving-api-fd65d7d6-jwws6"
Jul 01 18:11:28.003967 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:28.003882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22235a60-9ad3-4ad5-8a29-2377aad43187-tls-certs\") pod \"model-serving-api-fd65d7d6-jwws6\" (UID: \"22235a60-9ad3-4ad5-8a29-2377aad43187\") " pod="kserve/model-serving-api-fd65d7d6-jwws6"
Jul 01 18:11:28.004121 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:28.003971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgn2\" (UniqueName: \"kubernetes.io/projected/22235a60-9ad3-4ad5-8a29-2377aad43187-kube-api-access-hkgn2\") pod \"model-serving-api-fd65d7d6-jwws6\" (UID: \"22235a60-9ad3-4ad5-8a29-2377aad43187\") " pod="kserve/model-serving-api-fd65d7d6-jwws6"
Jul 01 18:11:28.006461 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:28.006432    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22235a60-9ad3-4ad5-8a29-2377aad43187-tls-certs\") pod \"model-serving-api-fd65d7d6-jwws6\" (UID: \"22235a60-9ad3-4ad5-8a29-2377aad43187\") " pod="kserve/model-serving-api-fd65d7d6-jwws6"
Jul 01 18:11:28.013071 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:28.013046    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkgn2\" (UniqueName: \"kubernetes.io/projected/22235a60-9ad3-4ad5-8a29-2377aad43187-kube-api-access-hkgn2\") pod \"model-serving-api-fd65d7d6-jwws6\" (UID: \"22235a60-9ad3-4ad5-8a29-2377aad43187\") " pod="kserve/model-serving-api-fd65d7d6-jwws6"
Jul 01 18:11:28.064649 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:28.064596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-jwws6"
Jul 01 18:11:28.201155 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:28.201116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-jwws6"]
Jul 01 18:11:28.203994 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:11:28.203965    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22235a60_9ad3_4ad5_8a29_2377aad43187.slice/crio-b480f33858a2aafde7f6a7c5548a7e60c0c727ce3e7c5889c5bcbd9d504e11e5 WatchSource:0}: Error finding container b480f33858a2aafde7f6a7c5548a7e60c0c727ce3e7c5889c5bcbd9d504e11e5: Status 404 returned error can't find the container with id b480f33858a2aafde7f6a7c5548a7e60c0c727ce3e7c5889c5bcbd9d504e11e5
Jul 01 18:11:28.205881 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:28.205866    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:11:28.448271 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:28.448239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-jwws6" event={"ID":"22235a60-9ad3-4ad5-8a29-2377aad43187","Type":"ContainerStarted","Data":"b480f33858a2aafde7f6a7c5548a7e60c0c727ce3e7c5889c5bcbd9d504e11e5"}
Jul 01 18:11:31.463726 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:31.463691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-jwws6" event={"ID":"22235a60-9ad3-4ad5-8a29-2377aad43187","Type":"ContainerStarted","Data":"4f5c0914fbe2d9a72c1c934b0dd3f65f6c825083b5d6e4b9fb1712341ed7724d"}
Jul 01 18:11:31.464126 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:31.463765    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-jwws6"
Jul 01 18:11:31.482251 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:31.482206    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-jwws6" podStartSLOduration=1.655267037 podStartE2EDuration="4.482190939s" totalImagesPullingTime="2.826923902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:11:31.48012442 +0000 UTC m=+322.042295056" watchObservedRunningTime="2026-07-01 18:11:31.482190939 +0000 UTC m=+322.044361574"
Jul 01 18:11:42.472116 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:42.472088    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-jwws6"
Jul 01 18:11:45.771052 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.771022    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-5kb5f"]
Jul 01 18:11:45.774448 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.774431    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5kb5f"
Jul 01 18:11:45.776847 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.776820    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 18:11:45.776847 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.776820    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wsph9\""
Jul 01 18:11:45.781453 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.781423    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5kb5f"]
Jul 01 18:11:45.853788 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.853758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-model-cache\") pod \"s3-init-5kb5f\" (UID: \"e1f01ea7-3fa7-4d90-b2c9-091c37b12550\") " pod="kserve/s3-init-5kb5f"
Jul 01 18:11:45.853946 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.853800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds7fg\" (UniqueName: \"kubernetes.io/projected/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-kube-api-access-ds7fg\") pod \"s3-init-5kb5f\" (UID: \"e1f01ea7-3fa7-4d90-b2c9-091c37b12550\") " pod="kserve/s3-init-5kb5f"
Jul 01 18:11:45.954993 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.954962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-model-cache\") pod \"s3-init-5kb5f\" (UID: \"e1f01ea7-3fa7-4d90-b2c9-091c37b12550\") " pod="kserve/s3-init-5kb5f"
Jul 01 18:11:45.955139 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.955006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds7fg\" (UniqueName: \"kubernetes.io/projected/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-kube-api-access-ds7fg\") pod \"s3-init-5kb5f\" (UID: \"e1f01ea7-3fa7-4d90-b2c9-091c37b12550\") " pod="kserve/s3-init-5kb5f"
Jul 01 18:11:45.955313 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.955296    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-model-cache\") pod \"s3-init-5kb5f\" (UID: \"e1f01ea7-3fa7-4d90-b2c9-091c37b12550\") " pod="kserve/s3-init-5kb5f"
Jul 01 18:11:45.963094 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:45.963071    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds7fg\" (UniqueName: \"kubernetes.io/projected/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-kube-api-access-ds7fg\") pod \"s3-init-5kb5f\" (UID: \"e1f01ea7-3fa7-4d90-b2c9-091c37b12550\") " pod="kserve/s3-init-5kb5f"
Jul 01 18:11:46.102190 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:46.102157    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5kb5f"
Jul 01 18:11:46.230522 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:46.230352    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5kb5f"]
Jul 01 18:11:46.233177 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:11:46.233144    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1f01ea7_3fa7_4d90_b2c9_091c37b12550.slice/crio-49e0ec2e9382bfb6acf291d2afb71738086640d8de035e111acd0a7fb9dbec56 WatchSource:0}: Error finding container 49e0ec2e9382bfb6acf291d2afb71738086640d8de035e111acd0a7fb9dbec56: Status 404 returned error can't find the container with id 49e0ec2e9382bfb6acf291d2afb71738086640d8de035e111acd0a7fb9dbec56
Jul 01 18:11:46.522902 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:46.522818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5kb5f" event={"ID":"e1f01ea7-3fa7-4d90-b2c9-091c37b12550","Type":"ContainerStarted","Data":"49e0ec2e9382bfb6acf291d2afb71738086640d8de035e111acd0a7fb9dbec56"}
Jul 01 18:11:53.556285 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:53.556243    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5kb5f" event={"ID":"e1f01ea7-3fa7-4d90-b2c9-091c37b12550","Type":"ContainerStarted","Data":"71ecbba722c03a20034556f21a02e3aa074f8157fcd0c7e8d531c34f70f9c4be"}
Jul 01 18:11:54.562178 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:54.562146    2577 generic.go:356] "Generic (PLEG): container finished" podID="e1f01ea7-3fa7-4d90-b2c9-091c37b12550" containerID="71ecbba722c03a20034556f21a02e3aa074f8157fcd0c7e8d531c34f70f9c4be" exitCode=0
Jul 01 18:11:54.562558 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:11:54.562231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5kb5f" event={"ID":"e1f01ea7-3fa7-4d90-b2c9-091c37b12550","Type":"ContainerDied","Data":"71ecbba722c03a20034556f21a02e3aa074f8157fcd0c7e8d531c34f70f9c4be"}
Jul 01 18:12:00.593240 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:12:00.593201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5kb5f" event={"ID":"e1f01ea7-3fa7-4d90-b2c9-091c37b12550","Type":"ContainerStarted","Data":"f8d1d8f7c2843a38c554babf0edcd12c6479b3ff32e1398cc3482daebfb839ab"}
Jul 01 18:12:00.610036 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:12:00.609982    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-5kb5f" podStartSLOduration=2.149357972 podStartE2EDuration="15.609960615s" totalImagesPullingTime="11.656885698s" totalInitContainerRuntime="1.803716945s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:11:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:12:00.608171682 +0000 UTC m=+351.170342319" watchObservedRunningTime="2026-07-01 18:12:00.609960615 +0000 UTC m=+351.172131251"
Jul 01 18:13:04.855469 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:04.855438    2577 generic.go:356] "Generic (PLEG): container finished" podID="e1f01ea7-3fa7-4d90-b2c9-091c37b12550" containerID="f8d1d8f7c2843a38c554babf0edcd12c6479b3ff32e1398cc3482daebfb839ab" exitCode=0
Jul 01 18:13:04.855836 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:04.855513    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5kb5f" event={"ID":"e1f01ea7-3fa7-4d90-b2c9-091c37b12550","Type":"ContainerDied","Data":"f8d1d8f7c2843a38c554babf0edcd12c6479b3ff32e1398cc3482daebfb839ab"}
Jul 01 18:13:05.987699 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:05.987672    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5kb5f"
Jul 01 18:13:06.072142 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:06.072111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-model-cache\") pod \"e1f01ea7-3fa7-4d90-b2c9-091c37b12550\" (UID: \"e1f01ea7-3fa7-4d90-b2c9-091c37b12550\") "
Jul 01 18:13:06.072316 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:06.072210    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds7fg\" (UniqueName: \"kubernetes.io/projected/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-kube-api-access-ds7fg\") pod \"e1f01ea7-3fa7-4d90-b2c9-091c37b12550\" (UID: \"e1f01ea7-3fa7-4d90-b2c9-091c37b12550\") "
Jul 01 18:13:06.074552 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:06.074519    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-kube-api-access-ds7fg" (OuterVolumeSpecName: "kube-api-access-ds7fg") pod "e1f01ea7-3fa7-4d90-b2c9-091c37b12550" (UID: "e1f01ea7-3fa7-4d90-b2c9-091c37b12550"). InnerVolumeSpecName "kube-api-access-ds7fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:13:06.090638 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:06.090595    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1f01ea7-3fa7-4d90-b2c9-091c37b12550" (UID: "e1f01ea7-3fa7-4d90-b2c9-091c37b12550"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:13:06.173179 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:06.173102    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-model-cache\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:13:06.173179 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:06.173133    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds7fg\" (UniqueName: \"kubernetes.io/projected/e1f01ea7-3fa7-4d90-b2c9-091c37b12550-kube-api-access-ds7fg\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:13:06.864919 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:06.864893    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5kb5f"
Jul 01 18:13:06.865104 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:06.864921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5kb5f" event={"ID":"e1f01ea7-3fa7-4d90-b2c9-091c37b12550","Type":"ContainerDied","Data":"49e0ec2e9382bfb6acf291d2afb71738086640d8de035e111acd0a7fb9dbec56"}
Jul 01 18:13:06.865104 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:06.864951    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e0ec2e9382bfb6acf291d2afb71738086640d8de035e111acd0a7fb9dbec56"
Jul 01 18:13:18.949089 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.949056    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"]
Jul 01 18:13:18.949523 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.949429    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f01ea7-3fa7-4d90-b2c9-091c37b12550" containerName="download-hf-model"
Jul 01 18:13:18.949523 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.949442    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f01ea7-3fa7-4d90-b2c9-091c37b12550" containerName="download-hf-model"
Jul 01 18:13:18.949523 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.949467    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f01ea7-3fa7-4d90-b2c9-091c37b12550" containerName="s3-init"
Jul 01 18:13:18.949523 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.949473    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f01ea7-3fa7-4d90-b2c9-091c37b12550" containerName="s3-init"
Jul 01 18:13:18.949670 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.949536    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f01ea7-3fa7-4d90-b2c9-091c37b12550" containerName="s3-init"
Jul 01 18:13:18.953214 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.953196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:18.955731 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.955710    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s5mjf\""
Jul 01 18:13:18.955868 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.955849    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:13:18.956090 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.956075    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a980c-kube-rbac-proxy-sar-config\""
Jul 01 18:13:18.956190 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.956103    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a980c-predictor-serving-cert\""
Jul 01 18:13:18.956253 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.956194    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 18:13:18.956309 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.956194    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 18:13:18.956366 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.956315    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:13:18.967382 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.967357    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"]
Jul 01 18:13:18.978000 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.977925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:18.978000 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.977955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e027644-5b9a-48ae-a225-f800e0d29195-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:18.978000 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.977980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4r5\" (UniqueName: \"kubernetes.io/projected/8e027644-5b9a-48ae-a225-f800e0d29195-kube-api-access-6v4r5\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:18.978214 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.978087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e027644-5b9a-48ae-a225-f800e0d29195-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:18.978214 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:18.978157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.079107 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.079064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.079107 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.079114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.079319 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.079282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e027644-5b9a-48ae-a225-f800e0d29195-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.079363 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.079329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4r5\" (UniqueName: \"kubernetes.io/projected/8e027644-5b9a-48ae-a225-f800e0d29195-kube-api-access-6v4r5\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.079439 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.079422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e027644-5b9a-48ae-a225-f800e0d29195-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.079474 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:13:19.079424    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-a980c-predictor-serving-cert" not found
Jul 01 18:13:19.079557 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:13:19.079545    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e027644-5b9a-48ae-a225-f800e0d29195-proxy-tls podName:8e027644-5b9a-48ae-a225-f800e0d29195 nodeName:}" failed. No retries permitted until 2026-07-01 18:13:19.579522088 +0000 UTC m=+430.141692719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8e027644-5b9a-48ae-a225-f800e0d29195-proxy-tls") pod "isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" (UID: "8e027644-5b9a-48ae-a225-f800e0d29195") : secret "isvc-raw-sklearn-batcher-a980c-predictor-serving-cert" not found
Jul 01 18:13:19.079827 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.079804    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e027644-5b9a-48ae-a225-f800e0d29195-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.079924 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.079907    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.079960 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.079931    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.088304 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.088275    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4r5\" (UniqueName: \"kubernetes.io/projected/8e027644-5b9a-48ae-a225-f800e0d29195-kube-api-access-6v4r5\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.582206 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.582165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e027644-5b9a-48ae-a225-f800e0d29195-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.584607 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.584580    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e027644-5b9a-48ae-a225-f800e0d29195-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:19.864330 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:19.864299    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:20.005359 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:20.005340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"]
Jul 01 18:13:20.007392 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:13:20.007363    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e027644_5b9a_48ae_a225_f800e0d29195.slice/crio-c5d7f1d379e1b9286bfda17055151c95d33625397d6dcd604f0224ad077a17d5 WatchSource:0}: Error finding container c5d7f1d379e1b9286bfda17055151c95d33625397d6dcd604f0224ad077a17d5: Status 404 returned error can't find the container with id c5d7f1d379e1b9286bfda17055151c95d33625397d6dcd604f0224ad077a17d5
Jul 01 18:13:20.928320 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:20.928286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" event={"ID":"8e027644-5b9a-48ae-a225-f800e0d29195","Type":"ContainerStarted","Data":"387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959"}
Jul 01 18:13:20.928472 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:20.928374    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" event={"ID":"8e027644-5b9a-48ae-a225-f800e0d29195","Type":"ContainerStarted","Data":"c5d7f1d379e1b9286bfda17055151c95d33625397d6dcd604f0224ad077a17d5"}
Jul 01 18:13:23.942812 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:23.942779    2577 generic.go:356] "Generic (PLEG): container finished" podID="8e027644-5b9a-48ae-a225-f800e0d29195" containerID="387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959" exitCode=0
Jul 01 18:13:23.943255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:23.942855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" event={"ID":"8e027644-5b9a-48ae-a225-f800e0d29195","Type":"ContainerDied","Data":"387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959"}
Jul 01 18:13:39.026775 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:39.026740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" event={"ID":"8e027644-5b9a-48ae-a225-f800e0d29195","Type":"ContainerStarted","Data":"3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95"}
Jul 01 18:13:42.042739 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:42.042654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" event={"ID":"8e027644-5b9a-48ae-a225-f800e0d29195","Type":"ContainerStarted","Data":"297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d"}
Jul 01 18:13:45.060008 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:45.059967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" event={"ID":"8e027644-5b9a-48ae-a225-f800e0d29195","Type":"ContainerStarted","Data":"4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620"}
Jul 01 18:13:45.060400 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:45.060250    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:45.060465 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:45.060417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:45.061491 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:45.061463    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:13:45.081823 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:45.081766    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podStartSLOduration=2.419126565 podStartE2EDuration="27.081752614s" totalImagesPullingTime="20.877362102s" totalInitContainerRuntime="3.785263947s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:13:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:13:45.080552671 +0000 UTC m=+455.642723309" watchObservedRunningTime="2026-07-01 18:13:45.081752614 +0000 UTC m=+455.643923314"
Jul 01 18:13:46.064901 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:46.064855    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:46.065370 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:46.064984    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:13:46.065909 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:46.065880    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:13:46.069719 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:46.069698    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:13:47.069740 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:47.069702    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:13:47.070203 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:47.070072    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:13:48.073965 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:48.073928    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:13:48.074351 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:48.074156    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:13:58.074209 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:58.074169    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:13:58.074657 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:13:58.074636    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:14:08.074299 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:08.074253    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:14:08.074817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:08.074753    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:14:18.074383 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:18.074341    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:14:18.074862 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:18.074758    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:14:28.074253 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:28.074214    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:14:28.074747 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:28.074642    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:14:38.074378 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:38.074328    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:14:38.074806 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:38.074717    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:14:48.074403 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:48.074345    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:14:48.074819 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:48.074731    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:14:58.074604 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:58.074574    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:14:58.075073 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:14:58.075058    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:15:04.107215 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.107136    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"]
Jul 01 18:15:04.107602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.107483    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" containerID="cri-o://3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95" gracePeriod=30
Jul 01 18:15:04.107602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.107553    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kube-rbac-proxy" containerID="cri-o://297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d" gracePeriod=30
Jul 01 18:15:04.107817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.107600    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" containerID="cri-o://4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620" gracePeriod=30
Jul 01 18:15:04.117748 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.117718    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"]
Jul 01 18:15:04.121093 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.121072    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.123336 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.123318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-ebf1b-predictor-serving-cert\""
Jul 01 18:15:04.124034 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.124015    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-ebf1b-kube-rbac-proxy-sar-config\""
Jul 01 18:15:04.140418 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.140387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"]
Jul 01 18:15:04.195550 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.195518    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"]
Jul 01 18:15:04.198092 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.198076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.200401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.200380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-ebf1b-predictor-serving-cert\""
Jul 01 18:15:04.200534 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.200380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-ebf1b-kube-rbac-proxy-sar-config\""
Jul 01 18:15:04.201399 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.201382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.201469 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.201417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9994897a-b72e-4cdc-b9f3-a77e709543c5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.201506 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.201466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9994897a-b72e-4cdc-b9f3-a77e709543c5-proxy-tls\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.201506 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.201482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.201565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.201504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbxtl\" (UniqueName: \"kubernetes.io/projected/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-api-access-nbxtl\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.201565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.201552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-proxy-tls\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.201658 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.201574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.201658 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.201608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.201719 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.201664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.201719 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.201705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xsmv\" (UniqueName: \"kubernetes.io/projected/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-api-access-8xsmv\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.209189 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.209164    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"]
Jul 01 18:15:04.302220 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.302424 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xsmv\" (UniqueName: \"kubernetes.io/projected/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-api-access-8xsmv\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.302424 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.302424 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9994897a-b72e-4cdc-b9f3-a77e709543c5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.302424 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9994897a-b72e-4cdc-b9f3-a77e709543c5-proxy-tls\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.302424 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.302424 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbxtl\" (UniqueName: \"kubernetes.io/projected/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-api-access-nbxtl\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.302766 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-proxy-tls\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.302766 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.302766 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.302766 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302685    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.302930 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.302832    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9994897a-b72e-4cdc-b9f3-a77e709543c5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.303153 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.303125    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.303327 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.303307    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.303393 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.303331    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.303549 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.303529    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.305131 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.305101    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9994897a-b72e-4cdc-b9f3-a77e709543c5-proxy-tls\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.305286 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.305268    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-proxy-tls\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.310555 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.310527    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xsmv\" (UniqueName: \"kubernetes.io/projected/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-api-access-8xsmv\") pod \"isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.310555 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.310538    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbxtl\" (UniqueName: \"kubernetes.io/projected/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-api-access-nbxtl\") pod \"isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.396947 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.396861    2577 generic.go:356] "Generic (PLEG): container finished" podID="8e027644-5b9a-48ae-a225-f800e0d29195" containerID="297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d" exitCode=2
Jul 01 18:15:04.396947 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.396934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" event={"ID":"8e027644-5b9a-48ae-a225-f800e0d29195","Type":"ContainerDied","Data":"297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d"}
Jul 01 18:15:04.434264 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.434227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:04.509017 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.508813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:04.572789 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.572691    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"]
Jul 01 18:15:04.574911 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:15:04.574878    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9994897a_b72e_4cdc_b9f3_a77e709543c5.slice/crio-588f67194a3c6b337af5c9b70d7ab1e0ed212a464101c01104254d357ec52777 WatchSource:0}: Error finding container 588f67194a3c6b337af5c9b70d7ab1e0ed212a464101c01104254d357ec52777: Status 404 returned error can't find the container with id 588f67194a3c6b337af5c9b70d7ab1e0ed212a464101c01104254d357ec52777
Jul 01 18:15:04.647852 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:04.647784    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"]
Jul 01 18:15:04.651281 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:15:04.651258    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337a6fc4_6f5e_4cb7_bf45_5fb46907b6b1.slice/crio-ae9adc40a3cbc0c97e6460ec4cc2ad23ce0d8ae73b04f0bf650fcca3abf061a1 WatchSource:0}: Error finding container ae9adc40a3cbc0c97e6460ec4cc2ad23ce0d8ae73b04f0bf650fcca3abf061a1: Status 404 returned error can't find the container with id ae9adc40a3cbc0c97e6460ec4cc2ad23ce0d8ae73b04f0bf650fcca3abf061a1
Jul 01 18:15:05.403356 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:05.403319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" event={"ID":"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1","Type":"ContainerStarted","Data":"da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f"}
Jul 01 18:15:05.403870 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:05.403377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" event={"ID":"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1","Type":"ContainerStarted","Data":"ae9adc40a3cbc0c97e6460ec4cc2ad23ce0d8ae73b04f0bf650fcca3abf061a1"}
Jul 01 18:15:05.404803 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:05.404778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" event={"ID":"9994897a-b72e-4cdc-b9f3-a77e709543c5","Type":"ContainerStarted","Data":"f8d3a82aec4bd51d7044e66accff49cffd73698126430a43f8221f1a74d939ee"}
Jul 01 18:15:05.404904 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:05.404826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" event={"ID":"9994897a-b72e-4cdc-b9f3-a77e709543c5","Type":"ContainerStarted","Data":"588f67194a3c6b337af5c9b70d7ab1e0ed212a464101c01104254d357ec52777"}
Jul 01 18:15:06.066085 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:06.066041    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 01 18:15:08.074901 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:08.074864    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:15:08.075253 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:08.075025    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:15:08.420626 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:08.420589    2577 generic.go:356] "Generic (PLEG): container finished" podID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerID="f8d3a82aec4bd51d7044e66accff49cffd73698126430a43f8221f1a74d939ee" exitCode=0
Jul 01 18:15:08.420786 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:08.420657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" event={"ID":"9994897a-b72e-4cdc-b9f3-a77e709543c5","Type":"ContainerDied","Data":"f8d3a82aec4bd51d7044e66accff49cffd73698126430a43f8221f1a74d939ee"}
Jul 01 18:15:08.422261 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:08.422242    2577 generic.go:356] "Generic (PLEG): container finished" podID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerID="da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f" exitCode=0
Jul 01 18:15:08.422350 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:08.422288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" event={"ID":"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1","Type":"ContainerDied","Data":"da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f"}
Jul 01 18:15:09.432186 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:09.432154    2577 generic.go:356] "Generic (PLEG): container finished" podID="8e027644-5b9a-48ae-a225-f800e0d29195" containerID="3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95" exitCode=0
Jul 01 18:15:09.432680 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:09.432229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" event={"ID":"8e027644-5b9a-48ae-a225-f800e0d29195","Type":"ContainerDied","Data":"3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95"}
Jul 01 18:15:09.434604 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:09.434568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" event={"ID":"9994897a-b72e-4cdc-b9f3-a77e709543c5","Type":"ContainerStarted","Data":"850ad090d6d2bf3527878de419017b1850d02b0726f96ae25197350ea2831916"}
Jul 01 18:15:09.434740 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:09.434661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" event={"ID":"9994897a-b72e-4cdc-b9f3-a77e709543c5","Type":"ContainerStarted","Data":"f66c928b778e09736313ca89afccd71ff6b21eecdca119fcd357fe8aa061d412"}
Jul 01 18:15:09.434901 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:09.434874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:09.454398 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:09.454319    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podStartSLOduration=1.713052518 podStartE2EDuration="5.454304037s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.741251519s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:15:09.453728402 +0000 UTC m=+540.015899062" watchObservedRunningTime="2026-07-01 18:15:09.454304037 +0000 UTC m=+540.016474672"
Jul 01 18:15:10.439675 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:10.439643    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:10.441021 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:10.440989    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:15:11.065701 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:11.065650    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 01 18:15:11.444405 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:11.444372    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:15:16.065529 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:16.065481    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 01 18:15:16.065995 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:16.065664    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:15:16.449413 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:16.449377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:15:16.450041 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:16.450006    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:15:18.074896 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:18.074861    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:15:18.075310 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:18.075034    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:15:21.065461 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:21.065417    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 01 18:15:26.065444 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:26.065392    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 01 18:15:26.450551 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:26.450514    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:15:28.074396 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:28.074356    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:15:28.074850 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:28.074501    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:15:28.075535 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:28.075506    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:15:28.075682 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:28.075647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:15:29.546016 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:29.545981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" event={"ID":"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1","Type":"ContainerStarted","Data":"8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b"}
Jul 01 18:15:29.546533 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:29.546054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" event={"ID":"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1","Type":"ContainerStarted","Data":"d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216"}
Jul 01 18:15:29.546533 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:29.546353    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:29.546533 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:29.546398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:29.547698 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:29.547673    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:15:29.566387 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:29.566330    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podStartSLOduration=1.347767624 podStartE2EDuration="25.566315024s" totalImagesPullingTime="20.568551231s" totalInitContainerRuntime="3.649996169s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:15:29.565317499 +0000 UTC m=+560.127488135" watchObservedRunningTime="2026-07-01 18:15:29.566315024 +0000 UTC m=+560.128485660"
Jul 01 18:15:30.550233 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:30.550199    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:15:31.065520 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:31.065477    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 01 18:15:34.269582 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.269558    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:15:34.391415 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.391314    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v4r5\" (UniqueName: \"kubernetes.io/projected/8e027644-5b9a-48ae-a225-f800e0d29195-kube-api-access-6v4r5\") pod \"8e027644-5b9a-48ae-a225-f800e0d29195\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") "
Jul 01 18:15:34.391415 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.391360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-kube-rbac-proxy-sar-config\") pod \"8e027644-5b9a-48ae-a225-f800e0d29195\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") "
Jul 01 18:15:34.391663 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.391421    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e027644-5b9a-48ae-a225-f800e0d29195-kserve-provision-location\") pod \"8e027644-5b9a-48ae-a225-f800e0d29195\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") "
Jul 01 18:15:34.391663 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.391450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e027644-5b9a-48ae-a225-f800e0d29195-proxy-tls\") pod \"8e027644-5b9a-48ae-a225-f800e0d29195\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") "
Jul 01 18:15:34.391663 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.391551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-cabundle-cert\") pod \"8e027644-5b9a-48ae-a225-f800e0d29195\" (UID: \"8e027644-5b9a-48ae-a225-f800e0d29195\") "
Jul 01 18:15:34.391870 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.391837    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e027644-5b9a-48ae-a225-f800e0d29195-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e027644-5b9a-48ae-a225-f800e0d29195" (UID: "8e027644-5b9a-48ae-a225-f800e0d29195"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:15:34.391936 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.391887    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e027644-5b9a-48ae-a225-f800e0d29195" (UID: "8e027644-5b9a-48ae-a225-f800e0d29195"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:15:34.391988 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.391948    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e027644-5b9a-48ae-a225-f800e0d29195" (UID: "8e027644-5b9a-48ae-a225-f800e0d29195"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:15:34.393817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.393787    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e027644-5b9a-48ae-a225-f800e0d29195-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e027644-5b9a-48ae-a225-f800e0d29195" (UID: "8e027644-5b9a-48ae-a225-f800e0d29195"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:15:34.393934 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.393861    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e027644-5b9a-48ae-a225-f800e0d29195-kube-api-access-6v4r5" (OuterVolumeSpecName: "kube-api-access-6v4r5") pod "8e027644-5b9a-48ae-a225-f800e0d29195" (UID: "8e027644-5b9a-48ae-a225-f800e0d29195"). InnerVolumeSpecName "kube-api-access-6v4r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:15:34.492396 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.492359    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:15:34.492396 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.492399    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v4r5\" (UniqueName: \"kubernetes.io/projected/8e027644-5b9a-48ae-a225-f800e0d29195-kube-api-access-6v4r5\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:15:34.492600 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.492413    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e027644-5b9a-48ae-a225-f800e0d29195-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:15:34.492600 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.492426    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e027644-5b9a-48ae-a225-f800e0d29195-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:15:34.492600 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.492439    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e027644-5b9a-48ae-a225-f800e0d29195-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:15:34.570378 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.570338    2577 generic.go:356] "Generic (PLEG): container finished" podID="8e027644-5b9a-48ae-a225-f800e0d29195" containerID="4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620" exitCode=0
Jul 01 18:15:34.570571 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.570417    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"
Jul 01 18:15:34.570571 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.570418    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" event={"ID":"8e027644-5b9a-48ae-a225-f800e0d29195","Type":"ContainerDied","Data":"4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620"}
Jul 01 18:15:34.570571 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.570486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5" event={"ID":"8e027644-5b9a-48ae-a225-f800e0d29195","Type":"ContainerDied","Data":"c5d7f1d379e1b9286bfda17055151c95d33625397d6dcd604f0224ad077a17d5"}
Jul 01 18:15:34.570571 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.570517    2577 scope.go:117] "RemoveContainer" containerID="4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620"
Jul 01 18:15:34.579862 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.579844    2577 scope.go:117] "RemoveContainer" containerID="297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d"
Jul 01 18:15:34.587608 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.587543    2577 scope.go:117] "RemoveContainer" containerID="3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95"
Jul 01 18:15:34.593591 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.593563    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"]
Jul 01 18:15:34.596208 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.596183    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5"]
Jul 01 18:15:34.597344 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.597326    2577 scope.go:117] "RemoveContainer" containerID="387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959"
Jul 01 18:15:34.605203 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.605181    2577 scope.go:117] "RemoveContainer" containerID="4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620"
Jul 01 18:15:34.605504 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:15:34.605484    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620\": container with ID starting with 4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620 not found: ID does not exist" containerID="4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620"
Jul 01 18:15:34.605565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.605517    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620"} err="failed to get container status \"4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620\": rpc error: code = NotFound desc = could not find container \"4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620\": container with ID starting with 4fd2bcc1cdf0133abe59b3ad5ad7cdb26178c4c4383a83a723639e9b21bf0620 not found: ID does not exist"
Jul 01 18:15:34.605565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.605537    2577 scope.go:117] "RemoveContainer" containerID="297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d"
Jul 01 18:15:34.605830 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:15:34.605816    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d\": container with ID starting with 297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d not found: ID does not exist" containerID="297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d"
Jul 01 18:15:34.605879 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.605835    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d"} err="failed to get container status \"297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d\": rpc error: code = NotFound desc = could not find container \"297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d\": container with ID starting with 297bd64b7555f6dcdfcad687fa048f83edfa601ab8765cb4e9b5472dc61ae34d not found: ID does not exist"
Jul 01 18:15:34.605879 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.605848    2577 scope.go:117] "RemoveContainer" containerID="3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95"
Jul 01 18:15:34.606052 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:15:34.606033    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95\": container with ID starting with 3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95 not found: ID does not exist" containerID="3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95"
Jul 01 18:15:34.606103 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.606056    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95"} err="failed to get container status \"3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95\": rpc error: code = NotFound desc = could not find container \"3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95\": container with ID starting with 3c6f5985cbcc74f40494f9ba7924b3adc3cc1db53f5ca6fc9d6c98ddd1518c95 not found: ID does not exist"
Jul 01 18:15:34.606103 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.606067    2577 scope.go:117] "RemoveContainer" containerID="387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959"
Jul 01 18:15:34.606275 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:15:34.606258    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959\": container with ID starting with 387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959 not found: ID does not exist" containerID="387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959"
Jul 01 18:15:34.606326 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:34.606284    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959"} err="failed to get container status \"387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959\": rpc error: code = NotFound desc = could not find container \"387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959\": container with ID starting with 387110dde7f3f881972882c7f13385b77fd5accbdad79d71d8e97637fdcd5959 not found: ID does not exist"
Jul 01 18:15:35.555232 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:35.555206    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:15:35.555804 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:35.555776    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:15:35.869418 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:35.869383    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" path="/var/lib/kubelet/pods/8e027644-5b9a-48ae-a225-f800e0d29195/volumes"
Jul 01 18:15:36.450397 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:36.450359    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:15:45.556467 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:45.556424    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:15:46.450693 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:46.450655    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:15:55.556154 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:55.556111    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:15:56.450073 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:15:56.450034    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:16:05.556526 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:05.556485    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:16:06.450382 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:06.450327    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:16:09.832721 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:09.832686    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:16:09.834186 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:09.834163    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:16:15.555992 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:15.555953    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:16:16.450489 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:16.450449    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:16:17.864751 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:17.864715    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:16:25.556425 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:25.556387    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:16:27.870458 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:27.870431    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:16:35.557179 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:35.557102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:16:54.362553 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.362514    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"]
Jul 01 18:16:54.363092 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.362839    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" containerID="cri-o://f66c928b778e09736313ca89afccd71ff6b21eecdca119fcd357fe8aa061d412" gracePeriod=30
Jul 01 18:16:54.363092 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.362894    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kube-rbac-proxy" containerID="cri-o://850ad090d6d2bf3527878de419017b1850d02b0726f96ae25197350ea2831916" gracePeriod=30
Jul 01 18:16:54.438334 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438300    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"]
Jul 01 18:16:54.438780 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438763    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container"
Jul 01 18:16:54.438875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438783    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container"
Jul 01 18:16:54.438875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438802    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent"
Jul 01 18:16:54.438875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438811    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent"
Jul 01 18:16:54.438875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438833    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="storage-initializer"
Jul 01 18:16:54.438875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438842    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="storage-initializer"
Jul 01 18:16:54.438875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438875    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kube-rbac-proxy"
Jul 01 18:16:54.439174 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438883    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kube-rbac-proxy"
Jul 01 18:16:54.439174 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438966    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="agent"
Jul 01 18:16:54.439174 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438979    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kserve-container"
Jul 01 18:16:54.439174 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.438992    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e027644-5b9a-48ae-a225-f800e0d29195" containerName="kube-rbac-proxy"
Jul 01 18:16:54.441254 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.441230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.443989 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.443965    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c9b93-kube-rbac-proxy-sar-config\""
Jul 01 18:16:54.444123 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.443966    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c9b93-predictor-serving-cert\""
Jul 01 18:16:54.453329 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.453288    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"]
Jul 01 18:16:54.500825 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.500785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.500979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.500837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.500979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.500863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.500979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.500901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.500979 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.500942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlsj\" (UniqueName: \"kubernetes.io/projected/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-api-access-7hlsj\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.512632 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.512579    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"]
Jul 01 18:16:54.516121 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.516095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.518721 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.518684    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c9b93-predictor-serving-cert\""
Jul 01 18:16:54.519005 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.518980    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c9b93-kube-rbac-proxy-sar-config\""
Jul 01 18:16:54.527600 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.527568    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"]
Jul 01 18:16:54.546444 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.546409    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"]
Jul 01 18:16:54.546716 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.546670    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container" containerID="cri-o://d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216" gracePeriod=30
Jul 01 18:16:54.546778 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.546754    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kube-rbac-proxy" containerID="cri-o://8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b" gracePeriod=30
Jul 01 18:16:54.602222 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.602185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79826905-847d-477d-b67c-25a7f824df7a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.602405 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.602237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlsj\" (UniqueName: \"kubernetes.io/projected/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-api-access-7hlsj\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.602405 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.602381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56dvv\" (UniqueName: \"kubernetes.io/projected/79826905-847d-477d-b67c-25a7f824df7a-kube-api-access-56dvv\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.602483 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.602442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.602483 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.602474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79826905-847d-477d-b67c-25a7f824df7a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.602549 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.602494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.602585 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.602568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.602686 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.602667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.602734 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.602712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.602769 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.602741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.603064 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.603028    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.603331 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.603311    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.603331 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.603323    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.605199 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.605176    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.609974 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.609951    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlsj\" (UniqueName: \"kubernetes.io/projected/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-api-access-7hlsj\") pod \"isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.703625 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.703572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79826905-847d-477d-b67c-25a7f824df7a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.704024 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.703654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56dvv\" (UniqueName: \"kubernetes.io/projected/79826905-847d-477d-b67c-25a7f824df7a-kube-api-access-56dvv\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.704024 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.703682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.704024 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.703706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79826905-847d-477d-b67c-25a7f824df7a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.704024 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.703726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.704359 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.704038    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79826905-847d-477d-b67c-25a7f824df7a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.704359 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.704353    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.704431 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.704388    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.706376 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.706351    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79826905-847d-477d-b67c-25a7f824df7a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.712082 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.712054    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56dvv\" (UniqueName: \"kubernetes.io/projected/79826905-847d-477d-b67c-25a7f824df7a-kube-api-access-56dvv\") pod \"isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.753547 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.753519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:54.829420 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.829387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:54.900212 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.900044    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"]
Jul 01 18:16:54.902389 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:16:54.902354    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6c4380_bcbd_4dd3_aca3_7812e1e4d829.slice/crio-5a77a4ad89bf7a8dc5771e47eb1e3495ee6c2edc7d4764bbd43d1c63c0fcf772 WatchSource:0}: Error finding container 5a77a4ad89bf7a8dc5771e47eb1e3495ee6c2edc7d4764bbd43d1c63c0fcf772: Status 404 returned error can't find the container with id 5a77a4ad89bf7a8dc5771e47eb1e3495ee6c2edc7d4764bbd43d1c63c0fcf772
Jul 01 18:16:54.950875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.950825    2577 generic.go:356] "Generic (PLEG): container finished" podID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerID="8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b" exitCode=2
Jul 01 18:16:54.950875 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.950862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" event={"ID":"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1","Type":"ContainerDied","Data":"8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b"}
Jul 01 18:16:54.952520 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.952494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" event={"ID":"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829","Type":"ContainerStarted","Data":"5a77a4ad89bf7a8dc5771e47eb1e3495ee6c2edc7d4764bbd43d1c63c0fcf772"}
Jul 01 18:16:54.955151 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.955069    2577 generic.go:356] "Generic (PLEG): container finished" podID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerID="850ad090d6d2bf3527878de419017b1850d02b0726f96ae25197350ea2831916" exitCode=2
Jul 01 18:16:54.955151 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.955107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" event={"ID":"9994897a-b72e-4cdc-b9f3-a77e709543c5","Type":"ContainerDied","Data":"850ad090d6d2bf3527878de419017b1850d02b0726f96ae25197350ea2831916"}
Jul 01 18:16:54.983276 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:54.983253    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"]
Jul 01 18:16:54.985725 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:16:54.985689    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79826905_847d_477d_b67c_25a7f824df7a.slice/crio-db468b30d01cebb3c1acf1bd54f05b80cd4a75421263eb21c798ad5ffcb9a46a WatchSource:0}: Error finding container db468b30d01cebb3c1acf1bd54f05b80cd4a75421263eb21c798ad5ffcb9a46a: Status 404 returned error can't find the container with id db468b30d01cebb3c1acf1bd54f05b80cd4a75421263eb21c798ad5ffcb9a46a
Jul 01 18:16:55.551523 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:55.551479    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 01 18:16:55.555855 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:55.555818    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:16:55.961071 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:55.961020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" event={"ID":"79826905-847d-477d-b67c-25a7f824df7a","Type":"ContainerStarted","Data":"cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e"}
Jul 01 18:16:55.961263 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:55.961105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" event={"ID":"79826905-847d-477d-b67c-25a7f824df7a","Type":"ContainerStarted","Data":"db468b30d01cebb3c1acf1bd54f05b80cd4a75421263eb21c798ad5ffcb9a46a"}
Jul 01 18:16:55.962388 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:55.962366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" event={"ID":"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829","Type":"ContainerStarted","Data":"7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b"}
Jul 01 18:16:56.444666 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:56.444607    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 01 18:16:57.864758 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:57.864716    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:16:58.500275 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.500248    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:16:58.540837 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.540794    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kserve-provision-location\") pod \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") "
Jul 01 18:16:58.540837 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.540848    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-cabundle-cert\") pod \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") "
Jul 01 18:16:58.541105 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.540878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbxtl\" (UniqueName: \"kubernetes.io/projected/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-api-access-nbxtl\") pod \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") "
Jul 01 18:16:58.541105 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.540910    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-proxy-tls\") pod \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") "
Jul 01 18:16:58.541105 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.540929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-rbac-proxy-sar-config\") pod \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\" (UID: \"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1\") "
Jul 01 18:16:58.541276 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.541154    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" (UID: "337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:16:58.541357 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.541317    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" (UID: "337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:16:58.541454 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.541324    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:16:58.541454 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.541339    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" (UID: "337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:16:58.543317 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.543291    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-api-access-nbxtl" (OuterVolumeSpecName: "kube-api-access-nbxtl") pod "337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" (UID: "337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1"). InnerVolumeSpecName "kube-api-access-nbxtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:16:58.543464 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.543351    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" (UID: "337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:16:58.645057 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.645018    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbxtl\" (UniqueName: \"kubernetes.io/projected/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-api-access-nbxtl\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:16:58.645057 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.645058    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:16:58.645285 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.645072    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:16:58.645285 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.645086    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:16:58.978040 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.978001    2577 generic.go:356] "Generic (PLEG): container finished" podID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerID="d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216" exitCode=0
Jul 01 18:16:58.978506 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.978066    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" event={"ID":"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1","Type":"ContainerDied","Data":"d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216"}
Jul 01 18:16:58.978506 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.978077    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"
Jul 01 18:16:58.978506 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.978128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc" event={"ID":"337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1","Type":"ContainerDied","Data":"ae9adc40a3cbc0c97e6460ec4cc2ad23ce0d8ae73b04f0bf650fcca3abf061a1"}
Jul 01 18:16:58.978506 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.978159    2577 scope.go:117] "RemoveContainer" containerID="8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b"
Jul 01 18:16:58.979863 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.979839    2577 generic.go:356] "Generic (PLEG): container finished" podID="79826905-847d-477d-b67c-25a7f824df7a" containerID="cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e" exitCode=0
Jul 01 18:16:58.979985 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.979918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" event={"ID":"79826905-847d-477d-b67c-25a7f824df7a","Type":"ContainerDied","Data":"cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e"}
Jul 01 18:16:58.981714 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.981688    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerID="7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b" exitCode=0
Jul 01 18:16:58.981793 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.981715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" event={"ID":"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829","Type":"ContainerDied","Data":"7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b"}
Jul 01 18:16:58.988217 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.988188    2577 scope.go:117] "RemoveContainer" containerID="d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216"
Jul 01 18:16:58.998428 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:58.998405    2577 scope.go:117] "RemoveContainer" containerID="da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f"
Jul 01 18:16:59.009181 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.009162    2577 scope.go:117] "RemoveContainer" containerID="8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b"
Jul 01 18:16:59.009472 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:16:59.009454    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b\": container with ID starting with 8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b not found: ID does not exist" containerID="8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b"
Jul 01 18:16:59.009519 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.009481    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b"} err="failed to get container status \"8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b\": rpc error: code = NotFound desc = could not find container \"8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b\": container with ID starting with 8e424ba73bc8fca9904c5d00bdf6268386208fd8fdaac58b58db7e35de8ccc1b not found: ID does not exist"
Jul 01 18:16:59.009519 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.009498    2577 scope.go:117] "RemoveContainer" containerID="d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216"
Jul 01 18:16:59.009739 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:16:59.009722    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216\": container with ID starting with d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216 not found: ID does not exist" containerID="d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216"
Jul 01 18:16:59.009776 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.009746    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216"} err="failed to get container status \"d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216\": rpc error: code = NotFound desc = could not find container \"d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216\": container with ID starting with d5e0bb845062a0cf6967c6f4dc47036360160519e382467e1e2831d2530ac216 not found: ID does not exist"
Jul 01 18:16:59.009776 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.009763    2577 scope.go:117] "RemoveContainer" containerID="da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f"
Jul 01 18:16:59.009989 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:16:59.009973    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f\": container with ID starting with da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f not found: ID does not exist" containerID="da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f"
Jul 01 18:16:59.010043 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.009993    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f"} err="failed to get container status \"da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f\": rpc error: code = NotFound desc = could not find container \"da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f\": container with ID starting with da45910e32e5d410afa3d764e220aed19f1649b33725e43a1473d2d21e79790f not found: ID does not exist"
Jul 01 18:16:59.034479 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.034447    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"]
Jul 01 18:16:59.038371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.038347    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc"]
Jul 01 18:16:59.869557 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.869526    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" path="/var/lib/kubelet/pods/337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1/volumes"
Jul 01 18:16:59.992388 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.992354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" event={"ID":"79826905-847d-477d-b67c-25a7f824df7a","Type":"ContainerStarted","Data":"2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223"}
Jul 01 18:16:59.992891 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.992444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" event={"ID":"79826905-847d-477d-b67c-25a7f824df7a","Type":"ContainerStarted","Data":"93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107"}
Jul 01 18:16:59.992891 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.992692    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:59.992891 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.992864    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:16:59.994219 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.994178    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:16:59.995348 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.995326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" event={"ID":"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829","Type":"ContainerStarted","Data":"adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77"}
Jul 01 18:16:59.995488 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.995387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" event={"ID":"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829","Type":"ContainerStarted","Data":"a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f"}
Jul 01 18:16:59.996344 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.996062    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:59.996344 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.996270    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:16:59.999748 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:16:59.999605    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:17:00.001375 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.001351    2577 generic.go:356] "Generic (PLEG): container finished" podID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerID="f66c928b778e09736313ca89afccd71ff6b21eecdca119fcd357fe8aa061d412" exitCode=0
Jul 01 18:17:00.001485 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.001414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" event={"ID":"9994897a-b72e-4cdc-b9f3-a77e709543c5","Type":"ContainerDied","Data":"f66c928b778e09736313ca89afccd71ff6b21eecdca119fcd357fe8aa061d412"}
Jul 01 18:17:00.017321 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.017259    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podStartSLOduration=2.398775759 podStartE2EDuration="6.017241597s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.618465838s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:17:00.014690433 +0000 UTC m=+650.576861088" watchObservedRunningTime="2026-07-01 18:17:00.017241597 +0000 UTC m=+650.579412232"
Jul 01 18:17:00.019238 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.019215    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:17:00.034538 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.034479    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podStartSLOduration=2.406929987 podStartE2EDuration="6.034457169s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.627527182s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:17:00.033545271 +0000 UTC m=+650.595715909" watchObservedRunningTime="2026-07-01 18:17:00.034457169 +0000 UTC m=+650.596627807"
Jul 01 18:17:00.162223 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.162145    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9994897a-b72e-4cdc-b9f3-a77e709543c5-proxy-tls\") pod \"9994897a-b72e-4cdc-b9f3-a77e709543c5\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") "
Jul 01 18:17:00.162223 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.162199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xsmv\" (UniqueName: \"kubernetes.io/projected/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-api-access-8xsmv\") pod \"9994897a-b72e-4cdc-b9f3-a77e709543c5\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") "
Jul 01 18:17:00.162450 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.162242    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9994897a-b72e-4cdc-b9f3-a77e709543c5-kserve-provision-location\") pod \"9994897a-b72e-4cdc-b9f3-a77e709543c5\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") "
Jul 01 18:17:00.162450 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.162278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-cabundle-cert\") pod \"9994897a-b72e-4cdc-b9f3-a77e709543c5\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") "
Jul 01 18:17:00.162450 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.162371    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-rbac-proxy-sar-config\") pod \"9994897a-b72e-4cdc-b9f3-a77e709543c5\" (UID: \"9994897a-b72e-4cdc-b9f3-a77e709543c5\") "
Jul 01 18:17:00.162641 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.162557    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9994897a-b72e-4cdc-b9f3-a77e709543c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9994897a-b72e-4cdc-b9f3-a77e709543c5" (UID: "9994897a-b72e-4cdc-b9f3-a77e709543c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:17:00.162698 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.162674    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9994897a-b72e-4cdc-b9f3-a77e709543c5" (UID: "9994897a-b72e-4cdc-b9f3-a77e709543c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:17:00.162748 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.162705    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9994897a-b72e-4cdc-b9f3-a77e709543c5-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:17:00.162748 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.162736    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9994897a-b72e-4cdc-b9f3-a77e709543c5" (UID: "9994897a-b72e-4cdc-b9f3-a77e709543c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:17:00.164519 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.164496    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9994897a-b72e-4cdc-b9f3-a77e709543c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9994897a-b72e-4cdc-b9f3-a77e709543c5" (UID: "9994897a-b72e-4cdc-b9f3-a77e709543c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:17:00.164519 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.164501    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-api-access-8xsmv" (OuterVolumeSpecName: "kube-api-access-8xsmv") pod "9994897a-b72e-4cdc-b9f3-a77e709543c5" (UID: "9994897a-b72e-4cdc-b9f3-a77e709543c5"). InnerVolumeSpecName "kube-api-access-8xsmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:17:00.263993 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.263948    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:17:00.263993 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.263983    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:17:00.263993 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.263993    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9994897a-b72e-4cdc-b9f3-a77e709543c5-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:17:00.263993 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:00.264001    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xsmv\" (UniqueName: \"kubernetes.io/projected/9994897a-b72e-4cdc-b9f3-a77e709543c5-kube-api-access-8xsmv\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:17:01.008998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:01.008971    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"
Jul 01 18:17:01.008998 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:01.008981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf" event={"ID":"9994897a-b72e-4cdc-b9f3-a77e709543c5","Type":"ContainerDied","Data":"588f67194a3c6b337af5c9b70d7ab1e0ed212a464101c01104254d357ec52777"}
Jul 01 18:17:01.009547 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:01.009048    2577 scope.go:117] "RemoveContainer" containerID="850ad090d6d2bf3527878de419017b1850d02b0726f96ae25197350ea2831916"
Jul 01 18:17:01.009547 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:01.009242    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:17:01.009903 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:01.009877    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:17:01.017892 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:01.017869    2577 scope.go:117] "RemoveContainer" containerID="f66c928b778e09736313ca89afccd71ff6b21eecdca119fcd357fe8aa061d412"
Jul 01 18:17:01.026671 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:01.026648    2577 scope.go:117] "RemoveContainer" containerID="f8d3a82aec4bd51d7044e66accff49cffd73698126430a43f8221f1a74d939ee"
Jul 01 18:17:01.031747 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:01.031715    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"]
Jul 01 18:17:01.036760 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:01.036736    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf"]
Jul 01 18:17:01.869861 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:01.869829    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" path="/var/lib/kubelet/pods/9994897a-b72e-4cdc-b9f3-a77e709543c5/volumes"
Jul 01 18:17:02.015181 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:02.015139    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:17:06.015044 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:06.015013    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:17:06.015513 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:06.015489    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:17:07.019244 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:07.019209    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:17:07.019813 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:07.019786    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:17:16.016162 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:16.016110    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:17:17.019831 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:17.019799    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:17:26.016097 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:26.016053    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:17:27.020009 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:27.019968    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:17:36.016427 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:36.016389    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:17:37.020037 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:37.019989    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:17:46.016078 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:46.016034    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:17:47.019866 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:47.019826    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:17:56.016010 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:56.015973    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:17:57.019829 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:17:57.019789    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:18:06.016892 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:06.016814    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:18:07.020750 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:07.020707    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:18:17.020767 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:17.020739    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:18:44.719506 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719473    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"]
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719865    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kube-rbac-proxy"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719878    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kube-rbac-proxy"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719888    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kube-rbac-proxy"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719893    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kube-rbac-proxy"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719901    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719907    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719920    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="storage-initializer"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719927    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="storage-initializer"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719940    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719946    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719955    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="storage-initializer"
Jul 01 18:18:44.720011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.719960    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="storage-initializer"
Jul 01 18:18:44.720371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.720020    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kserve-container"
Jul 01 18:18:44.720371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.720027    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kube-rbac-proxy"
Jul 01 18:18:44.720371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.720034    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="337a6fc4-6f5e-4cb7-bf45-5fb46907b6b1" containerName="kserve-container"
Jul 01 18:18:44.720371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.720039    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9994897a-b72e-4cdc-b9f3-a77e709543c5" containerName="kube-rbac-proxy"
Jul 01 18:18:44.722981 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.722964    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:44.725935 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.725904    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-dbad7-kube-rbac-proxy-sar-config\""
Jul 01 18:18:44.726235 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.726220    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-dbad7-predictor-serving-cert\""
Jul 01 18:18:44.732501 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.732473    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"]
Jul 01 18:18:44.774565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.774529    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"]
Jul 01 18:18:44.774873 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.774848    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" containerID="cri-o://a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f" gracePeriod=30
Jul 01 18:18:44.774973 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.774882    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kube-rbac-proxy" containerID="cri-o://adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77" gracePeriod=30
Jul 01 18:18:44.791267 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.791238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") " pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:44.791399 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.791282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kbmv\" (UniqueName: \"kubernetes.io/projected/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-api-access-8kbmv\") pod \"message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") " pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:44.791460 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.791407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/300b0375-12eb-4350-a0ea-45bffd450ae0-proxy-tls\") pod \"message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") " pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:44.827557 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.827519    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"]
Jul 01 18:18:44.827825 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.827799    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container" containerID="cri-o://93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107" gracePeriod=30
Jul 01 18:18:44.827917 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.827844    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kube-rbac-proxy" containerID="cri-o://2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223" gracePeriod=30
Jul 01 18:18:44.892371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.892330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") " pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:44.892371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.892376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kbmv\" (UniqueName: \"kubernetes.io/projected/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-api-access-8kbmv\") pod \"message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") " pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:44.892602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.892446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/300b0375-12eb-4350-a0ea-45bffd450ae0-proxy-tls\") pod \"message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") " pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:44.892679 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:18:44.892656    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-serving-cert: secret "message-dumper-raw-dbad7-predictor-serving-cert" not found
Jul 01 18:18:44.892741 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:18:44.892730    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300b0375-12eb-4350-a0ea-45bffd450ae0-proxy-tls podName:300b0375-12eb-4350-a0ea-45bffd450ae0 nodeName:}" failed. No retries permitted until 2026-07-01 18:18:45.392713468 +0000 UTC m=+755.954884083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/300b0375-12eb-4350-a0ea-45bffd450ae0-proxy-tls") pod "message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7" (UID: "300b0375-12eb-4350-a0ea-45bffd450ae0") : secret "message-dumper-raw-dbad7-predictor-serving-cert" not found
Jul 01 18:18:44.893124 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.893105    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") " pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:44.900939 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:44.900911    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kbmv\" (UniqueName: \"kubernetes.io/projected/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-api-access-8kbmv\") pod \"message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") " pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:45.398560 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:45.398523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/300b0375-12eb-4350-a0ea-45bffd450ae0-proxy-tls\") pod \"message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") " pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:45.401130 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:45.401109    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/300b0375-12eb-4350-a0ea-45bffd450ae0-proxy-tls\") pod \"message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") " pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:45.489136 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:45.489102    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerID="adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77" exitCode=2
Jul 01 18:18:45.489305 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:45.489183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" event={"ID":"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829","Type":"ContainerDied","Data":"adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77"}
Jul 01 18:18:45.491549 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:45.491525    2577 generic.go:356] "Generic (PLEG): container finished" podID="79826905-847d-477d-b67c-25a7f824df7a" containerID="2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223" exitCode=2
Jul 01 18:18:45.491692 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:45.491562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" event={"ID":"79826905-847d-477d-b67c-25a7f824df7a","Type":"ContainerDied","Data":"2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223"}
Jul 01 18:18:45.633923 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:45.633887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:45.765780 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:45.765757    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"]
Jul 01 18:18:45.768333 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:18:45.768303    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300b0375_12eb_4350_a0ea_45bffd450ae0.slice/crio-3658412b64f1c3d2bc73aee8391f4caa9e94c27b6cfde0059e643f7ff69acf8b WatchSource:0}: Error finding container 3658412b64f1c3d2bc73aee8391f4caa9e94c27b6cfde0059e643f7ff69acf8b: Status 404 returned error can't find the container with id 3658412b64f1c3d2bc73aee8391f4caa9e94c27b6cfde0059e643f7ff69acf8b
Jul 01 18:18:45.770367 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:45.770351    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:18:46.011081 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:46.010996    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 18:18:46.015701 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:46.015677    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:18:46.497439 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:46.497403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7" event={"ID":"300b0375-12eb-4350-a0ea-45bffd450ae0","Type":"ContainerStarted","Data":"3658412b64f1c3d2bc73aee8391f4caa9e94c27b6cfde0059e643f7ff69acf8b"}
Jul 01 18:18:47.015499 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:47.015463    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 18:18:47.020603 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:47.020579    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:18:47.503738 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:47.503703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7" event={"ID":"300b0375-12eb-4350-a0ea-45bffd450ae0","Type":"ContainerStarted","Data":"da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30"}
Jul 01 18:18:47.503899 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:47.503762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7" event={"ID":"300b0375-12eb-4350-a0ea-45bffd450ae0","Type":"ContainerStarted","Data":"bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54"}
Jul 01 18:18:47.503899 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:47.503791    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:47.503899 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:47.503820    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:47.505602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:47.505581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:18:47.525765 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:47.525720    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7" podStartSLOduration=2.323564346 podStartE2EDuration="3.525704998s" totalImagesPullingTime="1.202140652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:18:47.524348509 +0000 UTC m=+758.086519146" watchObservedRunningTime="2026-07-01 18:18:47.525704998 +0000 UTC m=+758.087875634"
Jul 01 18:18:48.693955 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.693930    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:18:48.827899 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.827801    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79826905-847d-477d-b67c-25a7f824df7a-proxy-tls\") pod \"79826905-847d-477d-b67c-25a7f824df7a\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") "
Jul 01 18:18:48.827899 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.827848    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79826905-847d-477d-b67c-25a7f824df7a-kserve-provision-location\") pod \"79826905-847d-477d-b67c-25a7f824df7a\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") "
Jul 01 18:18:48.827899 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.827867    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56dvv\" (UniqueName: \"kubernetes.io/projected/79826905-847d-477d-b67c-25a7f824df7a-kube-api-access-56dvv\") pod \"79826905-847d-477d-b67c-25a7f824df7a\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") "
Jul 01 18:18:48.828247 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.827912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-kube-rbac-proxy-sar-config\") pod \"79826905-847d-477d-b67c-25a7f824df7a\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") "
Jul 01 18:18:48.828247 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.827936    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-cabundle-cert\") pod \"79826905-847d-477d-b67c-25a7f824df7a\" (UID: \"79826905-847d-477d-b67c-25a7f824df7a\") "
Jul 01 18:18:48.828247 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.828161    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79826905-847d-477d-b67c-25a7f824df7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79826905-847d-477d-b67c-25a7f824df7a" (UID: "79826905-847d-477d-b67c-25a7f824df7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:18:48.828412 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.828254    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79826905-847d-477d-b67c-25a7f824df7a" (UID: "79826905-847d-477d-b67c-25a7f824df7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:18:48.828412 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.828314    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79826905-847d-477d-b67c-25a7f824df7a" (UID: "79826905-847d-477d-b67c-25a7f824df7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:18:48.830196 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.830168    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79826905-847d-477d-b67c-25a7f824df7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79826905-847d-477d-b67c-25a7f824df7a" (UID: "79826905-847d-477d-b67c-25a7f824df7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:18:48.830196 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.830170    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79826905-847d-477d-b67c-25a7f824df7a-kube-api-access-56dvv" (OuterVolumeSpecName: "kube-api-access-56dvv") pod "79826905-847d-477d-b67c-25a7f824df7a" (UID: "79826905-847d-477d-b67c-25a7f824df7a"). InnerVolumeSpecName "kube-api-access-56dvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:18:48.929553 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.929519    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:18:48.929553 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.929546    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79826905-847d-477d-b67c-25a7f824df7a-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:18:48.929553 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.929555    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79826905-847d-477d-b67c-25a7f824df7a-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:18:48.929553 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.929564    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56dvv\" (UniqueName: \"kubernetes.io/projected/79826905-847d-477d-b67c-25a7f824df7a-kube-api-access-56dvv\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:18:48.929819 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:48.929574    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79826905-847d-477d-b67c-25a7f824df7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:18:49.515201 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.515163    2577 generic.go:356] "Generic (PLEG): container finished" podID="79826905-847d-477d-b67c-25a7f824df7a" containerID="93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107" exitCode=0
Jul 01 18:18:49.515378 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.515220    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"
Jul 01 18:18:49.515378 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.515240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" event={"ID":"79826905-847d-477d-b67c-25a7f824df7a","Type":"ContainerDied","Data":"93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107"}
Jul 01 18:18:49.515378 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.515304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p" event={"ID":"79826905-847d-477d-b67c-25a7f824df7a","Type":"ContainerDied","Data":"db468b30d01cebb3c1acf1bd54f05b80cd4a75421263eb21c798ad5ffcb9a46a"}
Jul 01 18:18:49.515378 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.515341    2577 scope.go:117] "RemoveContainer" containerID="2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223"
Jul 01 18:18:49.524592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.524572    2577 scope.go:117] "RemoveContainer" containerID="93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107"
Jul 01 18:18:49.532165 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.532144    2577 scope.go:117] "RemoveContainer" containerID="cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e"
Jul 01 18:18:49.538802 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.538779    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"]
Jul 01 18:18:49.541093 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.541074    2577 scope.go:117] "RemoveContainer" containerID="2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223"
Jul 01 18:18:49.541392 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:18:49.541374    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223\": container with ID starting with 2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223 not found: ID does not exist" containerID="2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223"
Jul 01 18:18:49.541451 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.541401    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223"} err="failed to get container status \"2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223\": rpc error: code = NotFound desc = could not find container \"2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223\": container with ID starting with 2bba34c0a0ff24e46cef71c377c13d34794703146b07eddc2b1ef86a8f847223 not found: ID does not exist"
Jul 01 18:18:49.541451 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.541420    2577 scope.go:117] "RemoveContainer" containerID="93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107"
Jul 01 18:18:49.541686 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:18:49.541667    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107\": container with ID starting with 93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107 not found: ID does not exist" containerID="93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107"
Jul 01 18:18:49.541731 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.541695    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107"} err="failed to get container status \"93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107\": rpc error: code = NotFound desc = could not find container \"93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107\": container with ID starting with 93d45e8081fc7ac7775f01c5a2cae2bf6763ceba9dadcbacf32ce720f69a7107 not found: ID does not exist"
Jul 01 18:18:49.541731 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.541710    2577 scope.go:117] "RemoveContainer" containerID="cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e"
Jul 01 18:18:49.541987 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:18:49.541971    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e\": container with ID starting with cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e not found: ID does not exist" containerID="cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e"
Jul 01 18:18:49.542037 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.541995    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e"} err="failed to get container status \"cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e\": rpc error: code = NotFound desc = could not find container \"cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e\": container with ID starting with cd961217130c68f06af45c9e79667c4892df86a0f4cfb1b1f97aac28f5fcf23e not found: ID does not exist"
Jul 01 18:18:49.542543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.542524    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p"]
Jul 01 18:18:49.826866 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.826841    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:18:49.868791 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.868763    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79826905-847d-477d-b67c-25a7f824df7a" path="/var/lib/kubelet/pods/79826905-847d-477d-b67c-25a7f824df7a/volumes"
Jul 01 18:18:49.938996 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.938962    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-cabundle-cert\") pod \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") "
Jul 01 18:18:49.939188 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.939004    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-proxy-tls\") pod \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") "
Jul 01 18:18:49.939188 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.939025    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-rbac-proxy-sar-config\") pod \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") "
Jul 01 18:18:49.939188 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.939044    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hlsj\" (UniqueName: \"kubernetes.io/projected/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-api-access-7hlsj\") pod \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") "
Jul 01 18:18:49.939188 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.939097    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kserve-provision-location\") pod \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\" (UID: \"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829\") "
Jul 01 18:18:49.939461 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.939416    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" (UID: "ca6c4380-bcbd-4dd3-aca3-7812e1e4d829"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:18:49.939595 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.939498    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" (UID: "ca6c4380-bcbd-4dd3-aca3-7812e1e4d829"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:18:49.939595 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.939506    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" (UID: "ca6c4380-bcbd-4dd3-aca3-7812e1e4d829"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:18:49.941356 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.941332    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" (UID: "ca6c4380-bcbd-4dd3-aca3-7812e1e4d829"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:18:49.941452 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:49.941414    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-api-access-7hlsj" (OuterVolumeSpecName: "kube-api-access-7hlsj") pod "ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" (UID: "ca6c4380-bcbd-4dd3-aca3-7812e1e4d829"). InnerVolumeSpecName "kube-api-access-7hlsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:18:50.040332 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.040290    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:18:50.040332 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.040325    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:18:50.040332 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.040336    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:18:50.040332 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.040344    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:18:50.040601 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.040353    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlsj\" (UniqueName: \"kubernetes.io/projected/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829-kube-api-access-7hlsj\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:18:50.522818 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.522779    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerID="a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f" exitCode=0
Jul 01 18:18:50.522970 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.522844    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"
Jul 01 18:18:50.522970 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.522852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" event={"ID":"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829","Type":"ContainerDied","Data":"a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f"}
Jul 01 18:18:50.522970 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.522905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj" event={"ID":"ca6c4380-bcbd-4dd3-aca3-7812e1e4d829","Type":"ContainerDied","Data":"5a77a4ad89bf7a8dc5771e47eb1e3495ee6c2edc7d4764bbd43d1c63c0fcf772"}
Jul 01 18:18:50.522970 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.522938    2577 scope.go:117] "RemoveContainer" containerID="adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77"
Jul 01 18:18:50.531881 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.531863    2577 scope.go:117] "RemoveContainer" containerID="a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f"
Jul 01 18:18:50.539374 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.539352    2577 scope.go:117] "RemoveContainer" containerID="7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b"
Jul 01 18:18:50.546390 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.546363    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"]
Jul 01 18:18:50.548857 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.548839    2577 scope.go:117] "RemoveContainer" containerID="adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77"
Jul 01 18:18:50.549138 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:18:50.549120    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77\": container with ID starting with adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77 not found: ID does not exist" containerID="adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77"
Jul 01 18:18:50.549207 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.549151    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77"} err="failed to get container status \"adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77\": rpc error: code = NotFound desc = could not find container \"adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77\": container with ID starting with adcc4b1d2120dec7f051ad1e3d34f63517bd8fbf37e0e16b75900ca2b1c29d77 not found: ID does not exist"
Jul 01 18:18:50.549207 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.549178    2577 scope.go:117] "RemoveContainer" containerID="a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f"
Jul 01 18:18:50.549469 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:18:50.549447    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f\": container with ID starting with a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f not found: ID does not exist" containerID="a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f"
Jul 01 18:18:50.549507 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.549478    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f"} err="failed to get container status \"a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f\": rpc error: code = NotFound desc = could not find container \"a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f\": container with ID starting with a6eee21834e17a86132dc08f04f71091543bd571fde60f25d73c56eb5033dd5f not found: ID does not exist"
Jul 01 18:18:50.549507 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.549501    2577 scope.go:117] "RemoveContainer" containerID="7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b"
Jul 01 18:18:50.549717 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.549699    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj"]
Jul 01 18:18:50.549777 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:18:50.549727    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b\": container with ID starting with 7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b not found: ID does not exist" containerID="7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b"
Jul 01 18:18:50.549777 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:50.549755    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b"} err="failed to get container status \"7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b\": rpc error: code = NotFound desc = could not find container \"7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b\": container with ID starting with 7a5736014d0bfe0a3c9d432828316c821d3ce23501134258f70487f166bdac8b not found: ID does not exist"
Jul 01 18:18:51.869283 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:51.869253    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" path="/var/lib/kubelet/pods/ca6c4380-bcbd-4dd3-aca3-7812e1e4d829/volumes"
Jul 01 18:18:54.520602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:18:54.520573    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:19:04.768644 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.768593    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"]
Jul 01 18:19:04.768997 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.768944    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="storage-initializer"
Jul 01 18:19:04.768997 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.768955    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="storage-initializer"
Jul 01 18:19:04.768997 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.768972    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container"
Jul 01 18:19:04.768997 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.768978    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container"
Jul 01 18:19:04.768997 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.768984    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kube-rbac-proxy"
Jul 01 18:19:04.768997 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.768990    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kube-rbac-proxy"
Jul 01 18:19:04.769195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.769005    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="storage-initializer"
Jul 01 18:19:04.769195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.769011    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="storage-initializer"
Jul 01 18:19:04.769195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.769020    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kube-rbac-proxy"
Jul 01 18:19:04.769195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.769025    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kube-rbac-proxy"
Jul 01 18:19:04.769195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.769034    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container"
Jul 01 18:19:04.769195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.769039    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container"
Jul 01 18:19:04.769195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.769087    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kserve-container"
Jul 01 18:19:04.769195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.769094    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kube-rbac-proxy"
Jul 01 18:19:04.769195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.769101    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca6c4380-bcbd-4dd3-aca3-7812e1e4d829" containerName="kube-rbac-proxy"
Jul 01 18:19:04.769195 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.769108    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="79826905-847d-477d-b67c-25a7f824df7a" containerName="kserve-container"
Jul 01 18:19:04.772595 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.772572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.775411 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.775390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-dbad7-predictor-serving-cert\""
Jul 01 18:19:04.775411 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.775404    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-dbad7-kube-rbac-proxy-sar-config\""
Jul 01 18:19:04.775586 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.775395    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 18:19:04.775586 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.775395    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 18:19:04.782475 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.782449    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"]
Jul 01 18:19:04.873775 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.873738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsmzk\" (UniqueName: \"kubernetes.io/projected/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-api-access-zsmzk\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.873962 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.873785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.873962 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.873849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d415cd5a-da86-48ed-8e9f-4666a61d5542-proxy-tls\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.873962 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.873880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d415cd5a-da86-48ed-8e9f-4666a61d5542-kserve-provision-location\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.873962 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.873938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-cabundle-cert\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.974601 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.974560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.974805 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.974611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d415cd5a-da86-48ed-8e9f-4666a61d5542-proxy-tls\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.974805 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.974680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d415cd5a-da86-48ed-8e9f-4666a61d5542-kserve-provision-location\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.974805 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.974733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-cabundle-cert\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.974805 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:19:04.974756    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-serving-cert: secret "isvc-logger-raw-dbad7-predictor-serving-cert" not found
Jul 01 18:19:04.974805 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.974795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsmzk\" (UniqueName: \"kubernetes.io/projected/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-api-access-zsmzk\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.975089 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:19:04.974840    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d415cd5a-da86-48ed-8e9f-4666a61d5542-proxy-tls podName:d415cd5a-da86-48ed-8e9f-4666a61d5542 nodeName:}" failed. No retries permitted until 2026-07-01 18:19:05.474815365 +0000 UTC m=+776.036986009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d415cd5a-da86-48ed-8e9f-4666a61d5542-proxy-tls") pod "isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" (UID: "d415cd5a-da86-48ed-8e9f-4666a61d5542") : secret "isvc-logger-raw-dbad7-predictor-serving-cert" not found
Jul 01 18:19:04.975162 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.975144    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d415cd5a-da86-48ed-8e9f-4666a61d5542-kserve-provision-location\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.975388 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.975365    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.975457 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.975367    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-cabundle-cert\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:04.983787 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:04.983756    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsmzk\" (UniqueName: \"kubernetes.io/projected/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-api-access-zsmzk\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:05.479175 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:05.479119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d415cd5a-da86-48ed-8e9f-4666a61d5542-proxy-tls\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:05.481651 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:05.481605    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d415cd5a-da86-48ed-8e9f-4666a61d5542-proxy-tls\") pod \"isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") " pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:05.683554 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:05.683513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:05.822595 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:05.822531    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"]
Jul 01 18:19:05.825109 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:19:05.825083    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd415cd5a_da86_48ed_8e9f_4666a61d5542.slice/crio-2f9259e7754968f32d69fd4462c8c0c5c2c46621194c5fbcac2dbb5c8dfb5b02 WatchSource:0}: Error finding container 2f9259e7754968f32d69fd4462c8c0c5c2c46621194c5fbcac2dbb5c8dfb5b02: Status 404 returned error can't find the container with id 2f9259e7754968f32d69fd4462c8c0c5c2c46621194c5fbcac2dbb5c8dfb5b02
Jul 01 18:19:06.600388 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:06.600352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" event={"ID":"d415cd5a-da86-48ed-8e9f-4666a61d5542","Type":"ContainerStarted","Data":"8e4158a459d68364bd539906ff32b7a38649673dd1791e0d067c5e5792dea711"}
Jul 01 18:19:06.600572 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:06.600420    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" event={"ID":"d415cd5a-da86-48ed-8e9f-4666a61d5542","Type":"ContainerStarted","Data":"2f9259e7754968f32d69fd4462c8c0c5c2c46621194c5fbcac2dbb5c8dfb5b02"}
Jul 01 18:19:09.613911 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:09.613878    2577 generic.go:356] "Generic (PLEG): container finished" podID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerID="8e4158a459d68364bd539906ff32b7a38649673dd1791e0d067c5e5792dea711" exitCode=0
Jul 01 18:19:09.614369 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:09.613957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" event={"ID":"d415cd5a-da86-48ed-8e9f-4666a61d5542","Type":"ContainerDied","Data":"8e4158a459d68364bd539906ff32b7a38649673dd1791e0d067c5e5792dea711"}
Jul 01 18:19:10.621679 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:10.621644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" event={"ID":"d415cd5a-da86-48ed-8e9f-4666a61d5542","Type":"ContainerStarted","Data":"a74f12e474dca5e2b9c1d7239d8085ab86c782e778f90abb1e83e2724c797d8e"}
Jul 01 18:19:10.622112 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:10.621709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" event={"ID":"d415cd5a-da86-48ed-8e9f-4666a61d5542","Type":"ContainerStarted","Data":"ff5c311acb69e997cadb163758219f594cf239bab9dad1e5792154c302b59290"}
Jul 01 18:19:10.622112 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:10.621737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" event={"ID":"d415cd5a-da86-48ed-8e9f-4666a61d5542","Type":"ContainerStarted","Data":"a54797b4f4363e76e06b7b473b59c93e5e655a8d2dda22e12f87805d93ab4b18"}
Jul 01 18:19:10.622112 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:10.622024    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:10.622112 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:10.622078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:10.622112 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:10.622103    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:10.623492 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:10.623468    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:19:10.624164 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:10.624141    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:19:10.645171 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:10.645094    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podStartSLOduration=3.138206373 podStartE2EDuration="6.645074252s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.506867879s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:19:10.64478361 +0000 UTC m=+781.206954248" watchObservedRunningTime="2026-07-01 18:19:10.645074252 +0000 UTC m=+781.207244888"
Jul 01 18:19:11.626736 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:11.626688    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:19:11.627178 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:11.627158    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:19:16.630912 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:16.630884    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:19:16.631565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:16.631539    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:19:16.631771 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:16.631752    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:19:26.631478 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:26.631440    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:19:26.631927 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:26.631837    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:19:36.632197 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:36.632101    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:19:36.632604 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:36.632572    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:19:46.631787 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:46.631743    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:19:46.632316 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:46.632291    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:19:56.632371 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:56.632319    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:19:56.632827 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:19:56.632748    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:20:06.631428 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:06.631386    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:20:06.631907 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:06.631884    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:20:16.631587 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:16.631546    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:20:16.632025 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:16.632006    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:20:24.864834 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:24.864800    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:20:24.865369 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:24.864908    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:20:39.785885 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:39.785854    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7_300b0375-12eb-4350-a0ea-45bffd450ae0/kserve-container/0.log"
Jul 01 18:20:40.077255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.077178    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"]
Jul 01 18:20:40.080770 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.080746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.083401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.083380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-7a9f1-kube-rbac-proxy-sar-config\""
Jul 01 18:20:40.083401 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.083379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-7a9f1-predictor-serving-cert\""
Jul 01 18:20:40.091331 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.091301    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"]
Jul 01 18:20:40.091846 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.091815    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" containerID="cri-o://a54797b4f4363e76e06b7b473b59c93e5e655a8d2dda22e12f87805d93ab4b18" gracePeriod=30
Jul 01 18:20:40.092052 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.091874    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" containerID="cri-o://a74f12e474dca5e2b9c1d7239d8085ab86c782e778f90abb1e83e2724c797d8e" gracePeriod=30
Jul 01 18:20:40.092161 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.091879    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kube-rbac-proxy" containerID="cri-o://ff5c311acb69e997cadb163758219f594cf239bab9dad1e5792154c302b59290" gracePeriod=30
Jul 01 18:20:40.095916 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.095889    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"]
Jul 01 18:20:40.139969 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.139934    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"]
Jul 01 18:20:40.140261 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.140222    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7" podUID="300b0375-12eb-4350-a0ea-45bffd450ae0" containerName="kserve-container" containerID="cri-o://bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54" gracePeriod=30
Jul 01 18:20:40.140353 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.140305    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7" podUID="300b0375-12eb-4350-a0ea-45bffd450ae0" containerName="kube-rbac-proxy" containerID="cri-o://da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30" gracePeriod=30
Jul 01 18:20:40.220135 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.220098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/781a7847-d3da-4c6c-a614-6e87c6179fc6-proxy-tls\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.220135 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.220141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-cabundle-cert\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.220359 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.220236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk8sw\" (UniqueName: \"kubernetes.io/projected/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-api-access-xk8sw\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.220359 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.220304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/781a7847-d3da-4c6c-a614-6e87c6179fc6-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.220359 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.220343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.320923 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.320892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.321101 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.320933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/781a7847-d3da-4c6c-a614-6e87c6179fc6-proxy-tls\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.321101 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.320952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-cabundle-cert\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.321101 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.321000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk8sw\" (UniqueName: \"kubernetes.io/projected/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-api-access-xk8sw\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.321101 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.321044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/781a7847-d3da-4c6c-a614-6e87c6179fc6-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.321661 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.321596    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/781a7847-d3da-4c6c-a614-6e87c6179fc6-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.321796 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.321682    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.321860 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.321797    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-cabundle-cert\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.323740 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.323718    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/781a7847-d3da-4c6c-a614-6e87c6179fc6-proxy-tls\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.331012 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.330938    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk8sw\" (UniqueName: \"kubernetes.io/projected/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-api-access-xk8sw\") pod \"isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.385493 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.385463    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:20:40.394560 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.394529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:40.523506 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.523471    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-rbac-proxy-sar-config\") pod \"300b0375-12eb-4350-a0ea-45bffd450ae0\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") "
Jul 01 18:20:40.523737 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.523578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/300b0375-12eb-4350-a0ea-45bffd450ae0-proxy-tls\") pod \"300b0375-12eb-4350-a0ea-45bffd450ae0\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") "
Jul 01 18:20:40.523737 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.523639    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kbmv\" (UniqueName: \"kubernetes.io/projected/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-api-access-8kbmv\") pod \"300b0375-12eb-4350-a0ea-45bffd450ae0\" (UID: \"300b0375-12eb-4350-a0ea-45bffd450ae0\") "
Jul 01 18:20:40.523943 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.523916    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "300b0375-12eb-4350-a0ea-45bffd450ae0" (UID: "300b0375-12eb-4350-a0ea-45bffd450ae0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:20:40.525827 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.525802    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/300b0375-12eb-4350-a0ea-45bffd450ae0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "300b0375-12eb-4350-a0ea-45bffd450ae0" (UID: "300b0375-12eb-4350-a0ea-45bffd450ae0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:20:40.525892 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.525830    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-api-access-8kbmv" (OuterVolumeSpecName: "kube-api-access-8kbmv") pod "300b0375-12eb-4350-a0ea-45bffd450ae0" (UID: "300b0375-12eb-4350-a0ea-45bffd450ae0"). InnerVolumeSpecName "kube-api-access-8kbmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:20:40.535253 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.535188    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"]
Jul 01 18:20:40.537549 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:20:40.537524    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781a7847_d3da_4c6c_a614_6e87c6179fc6.slice/crio-4776d246c3a9bb97cd90a7c9434ff572b0c6e07c3ae02d782e7a9c6ccd0fc91c WatchSource:0}: Error finding container 4776d246c3a9bb97cd90a7c9434ff572b0c6e07c3ae02d782e7a9c6ccd0fc91c: Status 404 returned error can't find the container with id 4776d246c3a9bb97cd90a7c9434ff572b0c6e07c3ae02d782e7a9c6ccd0fc91c
Jul 01 18:20:40.624544 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.624502    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:20:40.624544 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.624535    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/300b0375-12eb-4350-a0ea-45bffd450ae0-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:20:40.624544 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:40.624547    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kbmv\" (UniqueName: \"kubernetes.io/projected/300b0375-12eb-4350-a0ea-45bffd450ae0-kube-api-access-8kbmv\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:20:41.047448 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.047408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" event={"ID":"781a7847-d3da-4c6c-a614-6e87c6179fc6","Type":"ContainerStarted","Data":"c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d"}
Jul 01 18:20:41.047916 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.047481    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" event={"ID":"781a7847-d3da-4c6c-a614-6e87c6179fc6","Type":"ContainerStarted","Data":"4776d246c3a9bb97cd90a7c9434ff572b0c6e07c3ae02d782e7a9c6ccd0fc91c"}
Jul 01 18:20:41.049147 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.049119    2577 generic.go:356] "Generic (PLEG): container finished" podID="300b0375-12eb-4350-a0ea-45bffd450ae0" containerID="da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30" exitCode=2
Jul 01 18:20:41.049147 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.049142    2577 generic.go:356] "Generic (PLEG): container finished" podID="300b0375-12eb-4350-a0ea-45bffd450ae0" containerID="bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54" exitCode=2
Jul 01 18:20:41.049326 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.049163    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"
Jul 01 18:20:41.049326 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.049197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7" event={"ID":"300b0375-12eb-4350-a0ea-45bffd450ae0","Type":"ContainerDied","Data":"da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30"}
Jul 01 18:20:41.049326 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.049251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7" event={"ID":"300b0375-12eb-4350-a0ea-45bffd450ae0","Type":"ContainerDied","Data":"bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54"}
Jul 01 18:20:41.049326 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.049281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7" event={"ID":"300b0375-12eb-4350-a0ea-45bffd450ae0","Type":"ContainerDied","Data":"3658412b64f1c3d2bc73aee8391f4caa9e94c27b6cfde0059e643f7ff69acf8b"}
Jul 01 18:20:41.049326 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.049315    2577 scope.go:117] "RemoveContainer" containerID="da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30"
Jul 01 18:20:41.052213 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.052184    2577 generic.go:356] "Generic (PLEG): container finished" podID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerID="ff5c311acb69e997cadb163758219f594cf239bab9dad1e5792154c302b59290" exitCode=2
Jul 01 18:20:41.052350 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.052251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" event={"ID":"d415cd5a-da86-48ed-8e9f-4666a61d5542","Type":"ContainerDied","Data":"ff5c311acb69e997cadb163758219f594cf239bab9dad1e5792154c302b59290"}
Jul 01 18:20:41.060017 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.059998    2577 scope.go:117] "RemoveContainer" containerID="bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54"
Jul 01 18:20:41.072468 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.072446    2577 scope.go:117] "RemoveContainer" containerID="da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30"
Jul 01 18:20:41.072841 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:20:41.072819    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30\": container with ID starting with da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30 not found: ID does not exist" containerID="da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30"
Jul 01 18:20:41.072929 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.072849    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30"} err="failed to get container status \"da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30\": rpc error: code = NotFound desc = could not find container \"da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30\": container with ID starting with da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30 not found: ID does not exist"
Jul 01 18:20:41.072929 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.072867    2577 scope.go:117] "RemoveContainer" containerID="bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54"
Jul 01 18:20:41.073149 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:20:41.073129    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54\": container with ID starting with bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54 not found: ID does not exist" containerID="bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54"
Jul 01 18:20:41.073193 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.073159    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54"} err="failed to get container status \"bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54\": rpc error: code = NotFound desc = could not find container \"bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54\": container with ID starting with bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54 not found: ID does not exist"
Jul 01 18:20:41.073193 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.073178    2577 scope.go:117] "RemoveContainer" containerID="da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30"
Jul 01 18:20:41.073452 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.073429    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30"} err="failed to get container status \"da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30\": rpc error: code = NotFound desc = could not find container \"da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30\": container with ID starting with da65a83e4a8ca1ada4520e8d257b7ec2bd10852d6383859756d5652ea002ad30 not found: ID does not exist"
Jul 01 18:20:41.073521 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.073454    2577 scope.go:117] "RemoveContainer" containerID="bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54"
Jul 01 18:20:41.073715 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.073697    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54"} err="failed to get container status \"bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54\": rpc error: code = NotFound desc = could not find container \"bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54\": container with ID starting with bd7e8183e5ee331fb7b162d4eca12ef66db5b7aa370e201deaf4473a0098fc54 not found: ID does not exist"
Jul 01 18:20:41.080334 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.080306    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"]
Jul 01 18:20:41.081839 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.081819    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7"]
Jul 01 18:20:41.627515 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.627474    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:20:41.869500 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:41.869465    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="300b0375-12eb-4350-a0ea-45bffd450ae0" path="/var/lib/kubelet/pods/300b0375-12eb-4350-a0ea-45bffd450ae0/volumes"
Jul 01 18:20:44.864320 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:44.864274    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:20:44.864774 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:44.864649    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:20:45.073075 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:45.073045    2577 generic.go:356] "Generic (PLEG): container finished" podID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerID="a54797b4f4363e76e06b7b473b59c93e5e655a8d2dda22e12f87805d93ab4b18" exitCode=0
Jul 01 18:20:45.073255 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:45.073104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" event={"ID":"d415cd5a-da86-48ed-8e9f-4666a61d5542","Type":"ContainerDied","Data":"a54797b4f4363e76e06b7b473b59c93e5e655a8d2dda22e12f87805d93ab4b18"}
Jul 01 18:20:45.074665 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:45.074641    2577 generic.go:356] "Generic (PLEG): container finished" podID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerID="c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d" exitCode=0
Jul 01 18:20:45.074797 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:45.074687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" event={"ID":"781a7847-d3da-4c6c-a614-6e87c6179fc6","Type":"ContainerDied","Data":"c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d"}
Jul 01 18:20:46.080963 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:46.080930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" event={"ID":"781a7847-d3da-4c6c-a614-6e87c6179fc6","Type":"ContainerStarted","Data":"70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d"}
Jul 01 18:20:46.081408 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:46.080992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" event={"ID":"781a7847-d3da-4c6c-a614-6e87c6179fc6","Type":"ContainerStarted","Data":"2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388"}
Jul 01 18:20:46.081408 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:46.081306    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:46.081494 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:46.081432    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:46.082759 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:46.082726    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:20:46.101677 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:46.101588    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podStartSLOduration=2.2767318530000002 podStartE2EDuration="6.10157095s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824839097s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:20:46.100388775 +0000 UTC m=+876.662559410" watchObservedRunningTime="2026-07-01 18:20:46.10157095 +0000 UTC m=+876.663741586"
Jul 01 18:20:46.626973 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:46.626934    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:20:47.085947 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:47.085854    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:20:51.627483 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:51.627435    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:20:51.627996 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:51.627586    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:20:52.090518 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:52.090427    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:20:52.091117 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:52.091084    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:20:54.865011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:54.864964    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:20:54.865435 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:54.865285    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:20:56.627448 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:20:56.627406    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:21:01.627253 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:01.627157    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:21:02.091904 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:02.091805    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:21:04.864356 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:04.864304    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:21:04.864817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:04.864449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:21:04.864817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:04.864596    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:21:04.864817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:04.864736    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:21:06.627178 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:06.627140    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:21:09.864095 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:09.864062    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:21:09.867723 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:09.867698    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:21:10.198598 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.198562    2577 generic.go:356] "Generic (PLEG): container finished" podID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerID="a74f12e474dca5e2b9c1d7239d8085ab86c782e778f90abb1e83e2724c797d8e" exitCode=0
Jul 01 18:21:10.198772 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.198649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" event={"ID":"d415cd5a-da86-48ed-8e9f-4666a61d5542","Type":"ContainerDied","Data":"a74f12e474dca5e2b9c1d7239d8085ab86c782e778f90abb1e83e2724c797d8e"}
Jul 01 18:21:10.249870 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.249843    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:21:10.375263 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.375174    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsmzk\" (UniqueName: \"kubernetes.io/projected/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-api-access-zsmzk\") pod \"d415cd5a-da86-48ed-8e9f-4666a61d5542\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") "
Jul 01 18:21:10.375263 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.375218    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d415cd5a-da86-48ed-8e9f-4666a61d5542-kserve-provision-location\") pod \"d415cd5a-da86-48ed-8e9f-4666a61d5542\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") "
Jul 01 18:21:10.375263 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.375253    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-cabundle-cert\") pod \"d415cd5a-da86-48ed-8e9f-4666a61d5542\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") "
Jul 01 18:21:10.375504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.375309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-rbac-proxy-sar-config\") pod \"d415cd5a-da86-48ed-8e9f-4666a61d5542\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") "
Jul 01 18:21:10.375504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.375346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d415cd5a-da86-48ed-8e9f-4666a61d5542-proxy-tls\") pod \"d415cd5a-da86-48ed-8e9f-4666a61d5542\" (UID: \"d415cd5a-da86-48ed-8e9f-4666a61d5542\") "
Jul 01 18:21:10.375670 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.375644    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d415cd5a-da86-48ed-8e9f-4666a61d5542-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d415cd5a-da86-48ed-8e9f-4666a61d5542" (UID: "d415cd5a-da86-48ed-8e9f-4666a61d5542"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:21:10.375744 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.375663    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d415cd5a-da86-48ed-8e9f-4666a61d5542" (UID: "d415cd5a-da86-48ed-8e9f-4666a61d5542"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:21:10.375744 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.375689    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d415cd5a-da86-48ed-8e9f-4666a61d5542" (UID: "d415cd5a-da86-48ed-8e9f-4666a61d5542"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:21:10.377482 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.377459    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-api-access-zsmzk" (OuterVolumeSpecName: "kube-api-access-zsmzk") pod "d415cd5a-da86-48ed-8e9f-4666a61d5542" (UID: "d415cd5a-da86-48ed-8e9f-4666a61d5542"). InnerVolumeSpecName "kube-api-access-zsmzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:21:10.377563 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.377528    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d415cd5a-da86-48ed-8e9f-4666a61d5542-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d415cd5a-da86-48ed-8e9f-4666a61d5542" (UID: "d415cd5a-da86-48ed-8e9f-4666a61d5542"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:21:10.477025 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.476979    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsmzk\" (UniqueName: \"kubernetes.io/projected/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-api-access-zsmzk\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:21:10.477025 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.477015    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d415cd5a-da86-48ed-8e9f-4666a61d5542-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:21:10.477025 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.477025    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:21:10.477025 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.477034    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d415cd5a-da86-48ed-8e9f-4666a61d5542-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:21:10.477299 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:10.477044    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d415cd5a-da86-48ed-8e9f-4666a61d5542-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:21:11.205232 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:11.205203    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"
Jul 01 18:21:11.205674 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:11.205208    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc" event={"ID":"d415cd5a-da86-48ed-8e9f-4666a61d5542","Type":"ContainerDied","Data":"2f9259e7754968f32d69fd4462c8c0c5c2c46621194c5fbcac2dbb5c8dfb5b02"}
Jul 01 18:21:11.205674 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:11.205368    2577 scope.go:117] "RemoveContainer" containerID="a74f12e474dca5e2b9c1d7239d8085ab86c782e778f90abb1e83e2724c797d8e"
Jul 01 18:21:11.214748 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:11.214701    2577 scope.go:117] "RemoveContainer" containerID="ff5c311acb69e997cadb163758219f594cf239bab9dad1e5792154c302b59290"
Jul 01 18:21:11.222836 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:11.222816    2577 scope.go:117] "RemoveContainer" containerID="a54797b4f4363e76e06b7b473b59c93e5e655a8d2dda22e12f87805d93ab4b18"
Jul 01 18:21:11.230643 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:11.229666    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"]
Jul 01 18:21:11.236481 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:11.236449    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc"]
Jul 01 18:21:11.237561 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:11.237543    2577 scope.go:117] "RemoveContainer" containerID="8e4158a459d68364bd539906ff32b7a38649673dd1791e0d067c5e5792dea711"
Jul 01 18:21:11.868862 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:11.868829    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" path="/var/lib/kubelet/pods/d415cd5a-da86-48ed-8e9f-4666a61d5542/volumes"
Jul 01 18:21:12.091519 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:12.091475    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:21:22.091968 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:22.091925    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:21:32.091430 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:32.091388    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:21:42.091842 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:42.091804    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:21:52.091804 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:21:52.091765    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:22:02.091051 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:22:02.091014    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:22:02.864907 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:22:02.864867    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:22:12.865407 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:22:12.865367    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:22:22.865939 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:22:22.865894    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:22:32.865499 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:22:32.865416    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:22:42.865783 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:22:42.865741    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:22:52.865017 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:22:52.864964    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:23:02.865446 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:02.865407    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:23:12.864902 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:12.864858    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:23:22.865771 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:22.865738    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:23:30.251702 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:30.251665    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"]
Jul 01 18:23:30.252289 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:30.251918    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" containerID="cri-o://2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388" gracePeriod=30
Jul 01 18:23:30.252289 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:30.251954    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kube-rbac-proxy" containerID="cri-o://70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d" gracePeriod=30
Jul 01 18:23:30.810688 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:30.810650    2577 generic.go:356] "Generic (PLEG): container finished" podID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerID="70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d" exitCode=2
Jul 01 18:23:30.810864 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:30.810708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" event={"ID":"781a7847-d3da-4c6c-a614-6e87c6179fc6","Type":"ContainerDied","Data":"70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d"}
Jul 01 18:23:32.086159 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:32.086105    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 01 18:23:32.865328 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:32.865284    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:23:35.421067 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421031    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"]
Jul 01 18:23:35.421449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421373    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="storage-initializer"
Jul 01 18:23:35.421449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421384    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="storage-initializer"
Jul 01 18:23:35.421449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421394    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="300b0375-12eb-4350-a0ea-45bffd450ae0" containerName="kube-rbac-proxy"
Jul 01 18:23:35.421449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421400    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="300b0375-12eb-4350-a0ea-45bffd450ae0" containerName="kube-rbac-proxy"
Jul 01 18:23:35.421449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421410    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kube-rbac-proxy"
Jul 01 18:23:35.421449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421415    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kube-rbac-proxy"
Jul 01 18:23:35.421449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421428    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container"
Jul 01 18:23:35.421449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421434    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container"
Jul 01 18:23:35.421449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421442    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent"
Jul 01 18:23:35.421449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421448    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent"
Jul 01 18:23:35.421781 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421457    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="300b0375-12eb-4350-a0ea-45bffd450ae0" containerName="kserve-container"
Jul 01 18:23:35.421781 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421462    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="300b0375-12eb-4350-a0ea-45bffd450ae0" containerName="kserve-container"
Jul 01 18:23:35.421781 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421520    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="agent"
Jul 01 18:23:35.421781 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421533    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="300b0375-12eb-4350-a0ea-45bffd450ae0" containerName="kube-rbac-proxy"
Jul 01 18:23:35.421781 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421541    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kserve-container"
Jul 01 18:23:35.421781 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421549    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d415cd5a-da86-48ed-8e9f-4666a61d5542" containerName="kube-rbac-proxy"
Jul 01 18:23:35.421781 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.421555    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="300b0375-12eb-4350-a0ea-45bffd450ae0" containerName="kserve-container"
Jul 01 18:23:35.424573 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.424552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.427000 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.426976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-72dc06-predictor-serving-cert\""
Jul 01 18:23:35.427134 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.427024    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-72dc06-kube-rbac-proxy-sar-config\""
Jul 01 18:23:35.438256 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.438224    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"]
Jul 01 18:23:35.523307 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.523262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2gl\" (UniqueName: \"kubernetes.io/projected/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-api-access-rl2gl\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.523307 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.523308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.523531 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.523339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kserve-provision-location\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.523531 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.523406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-cabundle-cert\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.523531 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.523443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cddfaabf-62a3-4671-af4d-a393ccfc78f9-proxy-tls\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.624189 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.624153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-cabundle-cert\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.624189 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.624195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cddfaabf-62a3-4671-af4d-a393ccfc78f9-proxy-tls\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.624484 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.624263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2gl\" (UniqueName: \"kubernetes.io/projected/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-api-access-rl2gl\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.624484 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.624286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.624484 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.624432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kserve-provision-location\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.624845 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.624818    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kserve-provision-location\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.624977 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.624957    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.625038 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.624975    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-cabundle-cert\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.626991 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.626972    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cddfaabf-62a3-4671-af4d-a393ccfc78f9-proxy-tls\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.631843 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.631821    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2gl\" (UniqueName: \"kubernetes.io/projected/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-api-access-rl2gl\") pod \"isvc-primary-72dc06-predictor-6cb65cb8db-9fztt\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") " pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.737282 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.737174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:35.875877 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:35.875853    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"]
Jul 01 18:23:35.878326 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:23:35.878290    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddfaabf_62a3_4671_af4d_a393ccfc78f9.slice/crio-5b39f0f5eb16d5690e93a86b9c3294737f978a01b3fa81dc56bc86cd7738f379 WatchSource:0}: Error finding container 5b39f0f5eb16d5690e93a86b9c3294737f978a01b3fa81dc56bc86cd7738f379: Status 404 returned error can't find the container with id 5b39f0f5eb16d5690e93a86b9c3294737f978a01b3fa81dc56bc86cd7738f379
Jul 01 18:23:36.837710 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:36.837659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" event={"ID":"cddfaabf-62a3-4671-af4d-a393ccfc78f9","Type":"ContainerStarted","Data":"fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8"}
Jul 01 18:23:36.838071 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:36.837730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" event={"ID":"cddfaabf-62a3-4671-af4d-a393ccfc78f9","Type":"ContainerStarted","Data":"5b39f0f5eb16d5690e93a86b9c3294737f978a01b3fa81dc56bc86cd7738f379"}
Jul 01 18:23:37.086521 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:37.086480    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 01 18:23:39.852739 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:39.852702    2577 generic.go:356] "Generic (PLEG): container finished" podID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerID="fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8" exitCode=0
Jul 01 18:23:39.853179 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:39.852773    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" event={"ID":"cddfaabf-62a3-4671-af4d-a393ccfc78f9","Type":"ContainerDied","Data":"fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8"}
Jul 01 18:23:40.503025 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.502990    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:23:40.570879 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.570791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk8sw\" (UniqueName: \"kubernetes.io/projected/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-api-access-xk8sw\") pod \"781a7847-d3da-4c6c-a614-6e87c6179fc6\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") "
Jul 01 18:23:40.570879 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.570842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-rbac-proxy-sar-config\") pod \"781a7847-d3da-4c6c-a614-6e87c6179fc6\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") "
Jul 01 18:23:40.570879 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.570881    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/781a7847-d3da-4c6c-a614-6e87c6179fc6-proxy-tls\") pod \"781a7847-d3da-4c6c-a614-6e87c6179fc6\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") "
Jul 01 18:23:40.571112 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.570966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/781a7847-d3da-4c6c-a614-6e87c6179fc6-kserve-provision-location\") pod \"781a7847-d3da-4c6c-a614-6e87c6179fc6\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") "
Jul 01 18:23:40.571112 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.571066    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-cabundle-cert\") pod \"781a7847-d3da-4c6c-a614-6e87c6179fc6\" (UID: \"781a7847-d3da-4c6c-a614-6e87c6179fc6\") "
Jul 01 18:23:40.571273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.571241    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781a7847-d3da-4c6c-a614-6e87c6179fc6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "781a7847-d3da-4c6c-a614-6e87c6179fc6" (UID: "781a7847-d3da-4c6c-a614-6e87c6179fc6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:23:40.571352 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.571244    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "781a7847-d3da-4c6c-a614-6e87c6179fc6" (UID: "781a7847-d3da-4c6c-a614-6e87c6179fc6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:23:40.571417 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.571399    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/781a7847-d3da-4c6c-a614-6e87c6179fc6-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:23:40.571417 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.571389    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "781a7847-d3da-4c6c-a614-6e87c6179fc6" (UID: "781a7847-d3da-4c6c-a614-6e87c6179fc6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:23:40.573171 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.573145    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-api-access-xk8sw" (OuterVolumeSpecName: "kube-api-access-xk8sw") pod "781a7847-d3da-4c6c-a614-6e87c6179fc6" (UID: "781a7847-d3da-4c6c-a614-6e87c6179fc6"). InnerVolumeSpecName "kube-api-access-xk8sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:23:40.573171 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.573164    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/781a7847-d3da-4c6c-a614-6e87c6179fc6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "781a7847-d3da-4c6c-a614-6e87c6179fc6" (UID: "781a7847-d3da-4c6c-a614-6e87c6179fc6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:23:40.672456 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.672418    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk8sw\" (UniqueName: \"kubernetes.io/projected/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-api-access-xk8sw\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:23:40.672456 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.672449    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:23:40.672456 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.672459    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/781a7847-d3da-4c6c-a614-6e87c6179fc6-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:23:40.672456 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.672468    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/781a7847-d3da-4c6c-a614-6e87c6179fc6-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:23:40.859844 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.859808    2577 generic.go:356] "Generic (PLEG): container finished" podID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerID="2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388" exitCode=0
Jul 01 18:23:40.860325 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.859876    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"
Jul 01 18:23:40.860325 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.859892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" event={"ID":"781a7847-d3da-4c6c-a614-6e87c6179fc6","Type":"ContainerDied","Data":"2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388"}
Jul 01 18:23:40.860325 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.859989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj" event={"ID":"781a7847-d3da-4c6c-a614-6e87c6179fc6","Type":"ContainerDied","Data":"4776d246c3a9bb97cd90a7c9434ff572b0c6e07c3ae02d782e7a9c6ccd0fc91c"}
Jul 01 18:23:40.860325 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.860023    2577 scope.go:117] "RemoveContainer" containerID="70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d"
Jul 01 18:23:40.862446 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.862417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" event={"ID":"cddfaabf-62a3-4671-af4d-a393ccfc78f9","Type":"ContainerStarted","Data":"441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3"}
Jul 01 18:23:40.862546 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.862465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" event={"ID":"cddfaabf-62a3-4671-af4d-a393ccfc78f9","Type":"ContainerStarted","Data":"09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045"}
Jul 01 18:23:40.862789 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.862754    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:40.862903 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.862812    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:40.864581 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.864540    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:23:40.869746 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.869731    2577 scope.go:117] "RemoveContainer" containerID="2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388"
Jul 01 18:23:40.877461 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.877437    2577 scope.go:117] "RemoveContainer" containerID="c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d"
Jul 01 18:23:40.883283 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.883239    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podStartSLOduration=2.267706973 podStartE2EDuration="5.88322276s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.615515787s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:23:40.882078298 +0000 UTC m=+1051.444248934" watchObservedRunningTime="2026-07-01 18:23:40.88322276 +0000 UTC m=+1051.445393395"
Jul 01 18:23:40.886727 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.886602    2577 scope.go:117] "RemoveContainer" containerID="70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d"
Jul 01 18:23:40.886962 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:23:40.886944    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d\": container with ID starting with 70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d not found: ID does not exist" containerID="70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d"
Jul 01 18:23:40.887030 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.886972    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d"} err="failed to get container status \"70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d\": rpc error: code = NotFound desc = could not find container \"70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d\": container with ID starting with 70632f7fd307636fcf0560593b22de125b59130169e9a9424196a604229a766d not found: ID does not exist"
Jul 01 18:23:40.887030 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.886991    2577 scope.go:117] "RemoveContainer" containerID="2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388"
Jul 01 18:23:40.887248 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:23:40.887234    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388\": container with ID starting with 2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388 not found: ID does not exist" containerID="2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388"
Jul 01 18:23:40.887294 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.887251    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388"} err="failed to get container status \"2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388\": rpc error: code = NotFound desc = could not find container \"2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388\": container with ID starting with 2520573296e0f44102a785d8a844a0cb1960279cb96f31c5b8b780f7adc85388 not found: ID does not exist"
Jul 01 18:23:40.887294 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.887263    2577 scope.go:117] "RemoveContainer" containerID="c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d"
Jul 01 18:23:40.887506 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:23:40.887492    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d\": container with ID starting with c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d not found: ID does not exist" containerID="c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d"
Jul 01 18:23:40.887566 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.887511    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d"} err="failed to get container status \"c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d\": rpc error: code = NotFound desc = could not find container \"c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d\": container with ID starting with c0cc0d320ba59173c72c37acd26b89a3ccacab6a85907f5537a5df8cf7cb1e9d not found: ID does not exist"
Jul 01 18:23:40.895712 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.895681    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"]
Jul 01 18:23:40.899268 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:40.899244    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj"]
Jul 01 18:23:41.869421 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:41.869380    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:23:41.869848 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:41.869541    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" path="/var/lib/kubelet/pods/781a7847-d3da-4c6c-a614-6e87c6179fc6/volumes"
Jul 01 18:23:46.873482 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:46.873451    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:23:46.873978 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:46.873954    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:23:56.874052 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:23:56.874013    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:24:06.874006 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:24:06.873915    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:24:16.874029 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:24:16.873986    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:24:26.873980 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:24:26.873943    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:24:36.874376 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:24:36.874336    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:24:46.874362 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:24:46.874315    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:24:56.875237 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:24:56.875207    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:25:05.595286 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.595247    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"]
Jul 01 18:25:05.595695 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.595602    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kube-rbac-proxy"
Jul 01 18:25:05.595695 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.595627    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kube-rbac-proxy"
Jul 01 18:25:05.595695 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.595638    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container"
Jul 01 18:25:05.595695 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.595645    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container"
Jul 01 18:25:05.595695 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.595659    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="storage-initializer"
Jul 01 18:25:05.595695 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.595665    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="storage-initializer"
Jul 01 18:25:05.595891 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.595728    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kserve-container"
Jul 01 18:25:05.595891 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.595735    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="781a7847-d3da-4c6c-a614-6e87c6179fc6" containerName="kube-rbac-proxy"
Jul 01 18:25:05.598941 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.598924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.601769 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.601736    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-72dc06-predictor-serving-cert\""
Jul 01 18:25:05.601908 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.601745    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-72dc06-kube-rbac-proxy-sar-config\""
Jul 01 18:25:05.601908 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.601778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-72dc06-dockercfg-lw82j\""
Jul 01 18:25:05.602045 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.601947    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-72dc06\""
Jul 01 18:25:05.607551 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.607518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"]
Jul 01 18:25:05.641308 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.641275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-cabundle-cert\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.641308 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.641314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kserve-provision-location\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.641517 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.641391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2ck\" (UniqueName: \"kubernetes.io/projected/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-api-access-pp2ck\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.641517 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.641418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-proxy-tls\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.641517 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.641436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.742714 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.742674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-cabundle-cert\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.742906 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.742722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kserve-provision-location\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.742906 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.742791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2ck\" (UniqueName: \"kubernetes.io/projected/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-api-access-pp2ck\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.742906 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.742880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-proxy-tls\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.743077 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.742915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.743217 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.743195    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kserve-provision-location\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.743433 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.743408    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-cabundle-cert\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.743583 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.743565    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.745478 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.745457    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-proxy-tls\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.750867 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.750846    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2ck\" (UniqueName: \"kubernetes.io/projected/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-api-access-pp2ck\") pod \"isvc-secondary-72dc06-predictor-6477d9654f-jlnkh\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") " pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:05.910248 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:05.910155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:06.045449 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:06.045427    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"]
Jul 01 18:25:06.048130 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:25:06.048103    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817dac3b_5c8a_4d5d_b0f3_febe0ade7c95.slice/crio-ed9387da43be2da8086e309e682eeea79b6fedd3d670c45346ae0c7dc15a34bb WatchSource:0}: Error finding container ed9387da43be2da8086e309e682eeea79b6fedd3d670c45346ae0c7dc15a34bb: Status 404 returned error can't find the container with id ed9387da43be2da8086e309e682eeea79b6fedd3d670c45346ae0c7dc15a34bb
Jul 01 18:25:06.232712 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:06.232607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh" event={"ID":"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95","Type":"ContainerStarted","Data":"49ce2a2236c3d9c248aab805539d0baa5bfc43714be58cf1c80b8b01f7dd82a9"}
Jul 01 18:25:06.232712 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:06.232688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh" event={"ID":"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95","Type":"ContainerStarted","Data":"ed9387da43be2da8086e309e682eeea79b6fedd3d670c45346ae0c7dc15a34bb"}
Jul 01 18:25:13.265861 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:13.265832    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_817dac3b-5c8a-4d5d-b0f3-febe0ade7c95/storage-initializer/0.log"
Jul 01 18:25:13.266225 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:13.265872    2577 generic.go:356] "Generic (PLEG): container finished" podID="817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" containerID="49ce2a2236c3d9c248aab805539d0baa5bfc43714be58cf1c80b8b01f7dd82a9" exitCode=1
Jul 01 18:25:13.266225 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:13.265931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh" event={"ID":"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95","Type":"ContainerDied","Data":"49ce2a2236c3d9c248aab805539d0baa5bfc43714be58cf1c80b8b01f7dd82a9"}
Jul 01 18:25:14.272353 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:14.272326    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_817dac3b-5c8a-4d5d-b0f3-febe0ade7c95/storage-initializer/0.log"
Jul 01 18:25:14.272747 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:14.272421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh" event={"ID":"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95","Type":"ContainerStarted","Data":"cafe053e3ee2785b9f7d937582a7f95b8ad18117fa730d6482d5ca47d3632d14"}
Jul 01 18:25:19.296009 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:19.295935    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_817dac3b-5c8a-4d5d-b0f3-febe0ade7c95/storage-initializer/1.log"
Jul 01 18:25:19.296410 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:19.296338    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_817dac3b-5c8a-4d5d-b0f3-febe0ade7c95/storage-initializer/0.log"
Jul 01 18:25:19.296410 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:19.296368    2577 generic.go:356] "Generic (PLEG): container finished" podID="817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" containerID="cafe053e3ee2785b9f7d937582a7f95b8ad18117fa730d6482d5ca47d3632d14" exitCode=1
Jul 01 18:25:19.296507 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:19.296419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh" event={"ID":"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95","Type":"ContainerDied","Data":"cafe053e3ee2785b9f7d937582a7f95b8ad18117fa730d6482d5ca47d3632d14"}
Jul 01 18:25:19.296507 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:19.296470    2577 scope.go:117] "RemoveContainer" containerID="49ce2a2236c3d9c248aab805539d0baa5bfc43714be58cf1c80b8b01f7dd82a9"
Jul 01 18:25:19.296868 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:19.296851    2577 scope.go:117] "RemoveContainer" containerID="49ce2a2236c3d9c248aab805539d0baa5bfc43714be58cf1c80b8b01f7dd82a9"
Jul 01 18:25:19.307529 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:25:19.307500    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_kserve-ci-e2e-test(817dac3b-5c8a-4d5d-b0f3-febe0ade7c95)\"" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh" podUID="817dac3b-5c8a-4d5d-b0f3-febe0ade7c95"
Jul 01 18:25:19.307667 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:25:19.307534    2577 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_kserve-ci-e2e-test_817dac3b-5c8a-4d5d-b0f3-febe0ade7c95_0 in pod sandbox ed9387da43be2da8086e309e682eeea79b6fedd3d670c45346ae0c7dc15a34bb from index: no such id: '49ce2a2236c3d9c248aab805539d0baa5bfc43714be58cf1c80b8b01f7dd82a9'" containerID="49ce2a2236c3d9c248aab805539d0baa5bfc43714be58cf1c80b8b01f7dd82a9"
Jul 01 18:25:19.307667 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:19.307569    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ce2a2236c3d9c248aab805539d0baa5bfc43714be58cf1c80b8b01f7dd82a9"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_kserve-ci-e2e-test_817dac3b-5c8a-4d5d-b0f3-febe0ade7c95_0 in pod sandbox ed9387da43be2da8086e309e682eeea79b6fedd3d670c45346ae0c7dc15a34bb from index: no such id: '49ce2a2236c3d9c248aab805539d0baa5bfc43714be58cf1c80b8b01f7dd82a9'"
Jul 01 18:25:20.302258 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:20.302232    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_817dac3b-5c8a-4d5d-b0f3-febe0ade7c95/storage-initializer/1.log"
Jul 01 18:25:23.613400 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.613363    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"]
Jul 01 18:25:23.668286 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.668254    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"]
Jul 01 18:25:23.668573 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.668498    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" containerID="cri-o://09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045" gracePeriod=30
Jul 01 18:25:23.668689 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.668558    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kube-rbac-proxy" containerID="cri-o://441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3" gracePeriod=30
Jul 01 18:25:23.790826 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.790803    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_817dac3b-5c8a-4d5d-b0f3-febe0ade7c95/storage-initializer/1.log"
Jul 01 18:25:23.792843 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.792824    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:23.909148 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.909060    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-rbac-proxy-sar-config\") pod \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") "
Jul 01 18:25:23.909148 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.909101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp2ck\" (UniqueName: \"kubernetes.io/projected/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-api-access-pp2ck\") pod \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") "
Jul 01 18:25:23.909148 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.909124    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-proxy-tls\") pod \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") "
Jul 01 18:25:23.909446 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.909159    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kserve-provision-location\") pod \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") "
Jul 01 18:25:23.909446 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.909195    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-cabundle-cert\") pod \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\" (UID: \"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95\") "
Jul 01 18:25:23.909589 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.909452    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" (UID: "817dac3b-5c8a-4d5d-b0f3-febe0ade7c95"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:25:23.909589 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.909536    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" (UID: "817dac3b-5c8a-4d5d-b0f3-febe0ade7c95"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:25:23.909723 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.909586    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:23.909723 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.909602    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" (UID: "817dac3b-5c8a-4d5d-b0f3-febe0ade7c95"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:25:23.911442 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.911419    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" (UID: "817dac3b-5c8a-4d5d-b0f3-febe0ade7c95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:25:23.911442 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:23.911417    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-api-access-pp2ck" (OuterVolumeSpecName: "kube-api-access-pp2ck") pod "817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" (UID: "817dac3b-5c8a-4d5d-b0f3-febe0ade7c95"). InnerVolumeSpecName "kube-api-access-pp2ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:25:24.011019 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.010982    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:24.011019 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.011011    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2ck\" (UniqueName: \"kubernetes.io/projected/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-kube-api-access-pp2ck\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:24.011019 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.011020    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:24.011019 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.011029    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:24.322061 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.321973    2577 generic.go:356] "Generic (PLEG): container finished" podID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerID="441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3" exitCode=2
Jul 01 18:25:24.322061 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.322047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" event={"ID":"cddfaabf-62a3-4671-af4d-a393ccfc78f9","Type":"ContainerDied","Data":"441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3"}
Jul 01 18:25:24.323244 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.323229    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_817dac3b-5c8a-4d5d-b0f3-febe0ade7c95/storage-initializer/1.log"
Jul 01 18:25:24.323309 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.323287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh" event={"ID":"817dac3b-5c8a-4d5d-b0f3-febe0ade7c95","Type":"ContainerDied","Data":"ed9387da43be2da8086e309e682eeea79b6fedd3d670c45346ae0c7dc15a34bb"}
Jul 01 18:25:24.323352 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.323313    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"
Jul 01 18:25:24.323352 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.323330    2577 scope.go:117] "RemoveContainer" containerID="cafe053e3ee2785b9f7d937582a7f95b8ad18117fa730d6482d5ca47d3632d14"
Jul 01 18:25:24.361496 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.361466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"]
Jul 01 18:25:24.367094 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:24.367067    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh"]
Jul 01 18:25:25.869465 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:25.869426    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" path="/var/lib/kubelet/pods/817dac3b-5c8a-4d5d-b0f3-febe0ade7c95/volumes"
Jul 01 18:25:26.869819 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:26.869780    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 01 18:25:26.874074 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:26.874042    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:25:28.525159 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.525134    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:25:28.653581 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.653546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kserve-provision-location\") pod \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") "
Jul 01 18:25:28.653817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.653590    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-rbac-proxy-sar-config\") pod \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") "
Jul 01 18:25:28.653817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.653648    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-cabundle-cert\") pod \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") "
Jul 01 18:25:28.653817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.653687    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cddfaabf-62a3-4671-af4d-a393ccfc78f9-proxy-tls\") pod \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") "
Jul 01 18:25:28.653817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.653766    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl2gl\" (UniqueName: \"kubernetes.io/projected/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-api-access-rl2gl\") pod \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\" (UID: \"cddfaabf-62a3-4671-af4d-a393ccfc78f9\") "
Jul 01 18:25:28.654025 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.653953    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cddfaabf-62a3-4671-af4d-a393ccfc78f9" (UID: "cddfaabf-62a3-4671-af4d-a393ccfc78f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:25:28.654074 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.654024    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cddfaabf-62a3-4671-af4d-a393ccfc78f9" (UID: "cddfaabf-62a3-4671-af4d-a393ccfc78f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:25:28.654074 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.654044    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cddfaabf-62a3-4671-af4d-a393ccfc78f9" (UID: "cddfaabf-62a3-4671-af4d-a393ccfc78f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:25:28.654141 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.654075    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:28.655985 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.655959    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddfaabf-62a3-4671-af4d-a393ccfc78f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cddfaabf-62a3-4671-af4d-a393ccfc78f9" (UID: "cddfaabf-62a3-4671-af4d-a393ccfc78f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:25:28.656073 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.656032    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-api-access-rl2gl" (OuterVolumeSpecName: "kube-api-access-rl2gl") pod "cddfaabf-62a3-4671-af4d-a393ccfc78f9" (UID: "cddfaabf-62a3-4671-af4d-a393ccfc78f9"). InnerVolumeSpecName "kube-api-access-rl2gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:25:28.754727 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.754688    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:28.754727 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.754719    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cddfaabf-62a3-4671-af4d-a393ccfc78f9-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:28.754727 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.754731    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cddfaabf-62a3-4671-af4d-a393ccfc78f9-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:28.755031 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.754739    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2gl\" (UniqueName: \"kubernetes.io/projected/cddfaabf-62a3-4671-af4d-a393ccfc78f9-kube-api-access-rl2gl\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:28.819000 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.818965    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"]
Jul 01 18:25:28.819316 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819303    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="storage-initializer"
Jul 01 18:25:28.819361 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819318    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="storage-initializer"
Jul 01 18:25:28.819361 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819332    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" containerName="storage-initializer"
Jul 01 18:25:28.819361 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819338    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" containerName="storage-initializer"
Jul 01 18:25:28.819361 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819347    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kube-rbac-proxy"
Jul 01 18:25:28.819361 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819353    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kube-rbac-proxy"
Jul 01 18:25:28.819361 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819363    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container"
Jul 01 18:25:28.819539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819368    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container"
Jul 01 18:25:28.819539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819381    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" containerName="storage-initializer"
Jul 01 18:25:28.819539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819386    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" containerName="storage-initializer"
Jul 01 18:25:28.819539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819436    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" containerName="storage-initializer"
Jul 01 18:25:28.819539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819444    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kube-rbac-proxy"
Jul 01 18:25:28.819539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819450    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerName="kserve-container"
Jul 01 18:25:28.819751 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.819545    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="817dac3b-5c8a-4d5d-b0f3-febe0ade7c95" containerName="storage-initializer"
Jul 01 18:25:28.822871 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.822853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:28.825241 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.825216    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b713a0-predictor-serving-cert\""
Jul 01 18:25:28.825402 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.825276    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-b713a0\""
Jul 01 18:25:28.825491 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.825477    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-b713a0-dockercfg-p47xs\""
Jul 01 18:25:28.825602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.825584    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b713a0-kube-rbac-proxy-sar-config\""
Jul 01 18:25:28.833111 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.833083    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"]
Jul 01 18:25:28.957165 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.957072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-cabundle-cert\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:28.957165 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.957123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/625afa52-f752-4d0a-9a4b-318a66355400-proxy-tls\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:28.957165 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.957154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbcb\" (UniqueName: \"kubernetes.io/projected/625afa52-f752-4d0a-9a4b-318a66355400-kube-api-access-cnbcb\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:28.957413 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.957178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:28.957413 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:28.957208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/625afa52-f752-4d0a-9a4b-318a66355400-kserve-provision-location\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.057949 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.057912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.058140 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.057964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/625afa52-f752-4d0a-9a4b-318a66355400-kserve-provision-location\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.058140 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.058006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-cabundle-cert\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.058140 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.058038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/625afa52-f752-4d0a-9a4b-318a66355400-proxy-tls\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.058140 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.058054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbcb\" (UniqueName: \"kubernetes.io/projected/625afa52-f752-4d0a-9a4b-318a66355400-kube-api-access-cnbcb\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.058490 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.058464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/625afa52-f752-4d0a-9a4b-318a66355400-kserve-provision-location\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.058650 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.058603    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.058775 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.058754    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-cabundle-cert\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.060695 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.060679    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/625afa52-f752-4d0a-9a4b-318a66355400-proxy-tls\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.067309 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.067283    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbcb\" (UniqueName: \"kubernetes.io/projected/625afa52-f752-4d0a-9a4b-318a66355400-kube-api-access-cnbcb\") pod \"isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") " pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.135144 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.135104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:29.271467 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.271432    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"]
Jul 01 18:25:29.274754 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:25:29.274725    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625afa52_f752_4d0a_9a4b_318a66355400.slice/crio-54f5b0825d468600797642801add1cd3381de221caadf2578e8d38461e6b1674 WatchSource:0}: Error finding container 54f5b0825d468600797642801add1cd3381de221caadf2578e8d38461e6b1674: Status 404 returned error can't find the container with id 54f5b0825d468600797642801add1cd3381de221caadf2578e8d38461e6b1674
Jul 01 18:25:29.347591 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.347562    2577 generic.go:356] "Generic (PLEG): container finished" podID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" containerID="09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045" exitCode=0
Jul 01 18:25:29.347732 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.347641    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"
Jul 01 18:25:29.347732 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.347641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" event={"ID":"cddfaabf-62a3-4671-af4d-a393ccfc78f9","Type":"ContainerDied","Data":"09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045"}
Jul 01 18:25:29.347732 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.347716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt" event={"ID":"cddfaabf-62a3-4671-af4d-a393ccfc78f9","Type":"ContainerDied","Data":"5b39f0f5eb16d5690e93a86b9c3294737f978a01b3fa81dc56bc86cd7738f379"}
Jul 01 18:25:29.347896 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.347754    2577 scope.go:117] "RemoveContainer" containerID="441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3"
Jul 01 18:25:29.349343 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.349317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb" event={"ID":"625afa52-f752-4d0a-9a4b-318a66355400","Type":"ContainerStarted","Data":"f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f"}
Jul 01 18:25:29.349343 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.349373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb" event={"ID":"625afa52-f752-4d0a-9a4b-318a66355400","Type":"ContainerStarted","Data":"54f5b0825d468600797642801add1cd3381de221caadf2578e8d38461e6b1674"}
Jul 01 18:25:29.357153 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.357116    2577 scope.go:117] "RemoveContainer" containerID="09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045"
Jul 01 18:25:29.365572 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.365549    2577 scope.go:117] "RemoveContainer" containerID="fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8"
Jul 01 18:25:29.374895 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.374877    2577 scope.go:117] "RemoveContainer" containerID="441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3"
Jul 01 18:25:29.375228 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:25:29.375198    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3\": container with ID starting with 441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3 not found: ID does not exist" containerID="441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3"
Jul 01 18:25:29.375320 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.375232    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3"} err="failed to get container status \"441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3\": rpc error: code = NotFound desc = could not find container \"441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3\": container with ID starting with 441e48478061e797933f6e37908563e6bd9be1a6c27c011e47e931daab7b6dd3 not found: ID does not exist"
Jul 01 18:25:29.375320 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.375257    2577 scope.go:117] "RemoveContainer" containerID="09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045"
Jul 01 18:25:29.375490 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:25:29.375471    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045\": container with ID starting with 09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045 not found: ID does not exist" containerID="09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045"
Jul 01 18:25:29.375532 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.375496    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045"} err="failed to get container status \"09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045\": rpc error: code = NotFound desc = could not find container \"09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045\": container with ID starting with 09a5cf4f3822b390d7903b6501bed0b4915ebff579b817fddd9ec7db7d992045 not found: ID does not exist"
Jul 01 18:25:29.375532 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.375511    2577 scope.go:117] "RemoveContainer" containerID="fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8"
Jul 01 18:25:29.375789 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:25:29.375774    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8\": container with ID starting with fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8 not found: ID does not exist" containerID="fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8"
Jul 01 18:25:29.375848 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.375792    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8"} err="failed to get container status \"fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8\": rpc error: code = NotFound desc = could not find container \"fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8\": container with ID starting with fc622ed876a6fac282ed60200d36d6fb0dfd5c5729efb2d82025c10e29839af8 not found: ID does not exist"
Jul 01 18:25:29.382232 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.382206    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"]
Jul 01 18:25:29.386584 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.386560    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt"]
Jul 01 18:25:29.875833 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:29.875798    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cddfaabf-62a3-4671-af4d-a393ccfc78f9" path="/var/lib/kubelet/pods/cddfaabf-62a3-4671-af4d-a393ccfc78f9/volumes"
Jul 01 18:25:33.370671 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:33.370643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb_625afa52-f752-4d0a-9a4b-318a66355400/storage-initializer/0.log"
Jul 01 18:25:33.371046 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:33.370683    2577 generic.go:356] "Generic (PLEG): container finished" podID="625afa52-f752-4d0a-9a4b-318a66355400" containerID="f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f" exitCode=1
Jul 01 18:25:33.371046 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:33.370767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb" event={"ID":"625afa52-f752-4d0a-9a4b-318a66355400","Type":"ContainerDied","Data":"f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f"}
Jul 01 18:25:33.816896 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:33.816793    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"]
Jul 01 18:25:34.376591 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:34.376563    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb_625afa52-f752-4d0a-9a4b-318a66355400/storage-initializer/0.log"
Jul 01 18:25:34.377011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:34.376649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb" event={"ID":"625afa52-f752-4d0a-9a4b-318a66355400","Type":"ContainerStarted","Data":"743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a"}
Jul 01 18:25:34.377011 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:34.376758    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb" podUID="625afa52-f752-4d0a-9a4b-318a66355400" containerName="storage-initializer" containerID="cri-o://743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a" gracePeriod=30
Jul 01 18:25:38.323348 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.323322    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb_625afa52-f752-4d0a-9a4b-318a66355400/storage-initializer/1.log"
Jul 01 18:25:38.323803 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.323788    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb_625afa52-f752-4d0a-9a4b-318a66355400/storage-initializer/0.log"
Jul 01 18:25:38.325855 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.325838    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:38.397054 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.397019    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb_625afa52-f752-4d0a-9a4b-318a66355400/storage-initializer/1.log"
Jul 01 18:25:38.397476 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.397457    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb_625afa52-f752-4d0a-9a4b-318a66355400/storage-initializer/0.log"
Jul 01 18:25:38.397694 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.397496    2577 generic.go:356] "Generic (PLEG): container finished" podID="625afa52-f752-4d0a-9a4b-318a66355400" containerID="743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a" exitCode=1
Jul 01 18:25:38.397694 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.397563    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"
Jul 01 18:25:38.397694 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.397664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb" event={"ID":"625afa52-f752-4d0a-9a4b-318a66355400","Type":"ContainerDied","Data":"743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a"}
Jul 01 18:25:38.397889 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.397741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb" event={"ID":"625afa52-f752-4d0a-9a4b-318a66355400","Type":"ContainerDied","Data":"54f5b0825d468600797642801add1cd3381de221caadf2578e8d38461e6b1674"}
Jul 01 18:25:38.397889 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.397780    2577 scope.go:117] "RemoveContainer" containerID="743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a"
Jul 01 18:25:38.406388 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.406370    2577 scope.go:117] "RemoveContainer" containerID="f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f"
Jul 01 18:25:38.414556 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.414537    2577 scope.go:117] "RemoveContainer" containerID="743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a"
Jul 01 18:25:38.414893 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:25:38.414859    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a\": container with ID starting with 743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a not found: ID does not exist" containerID="743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a"
Jul 01 18:25:38.414966 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.414904    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a"} err="failed to get container status \"743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a\": rpc error: code = NotFound desc = could not find container \"743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a\": container with ID starting with 743ee837d181aaa64b53bddde208fd1a40a3ec36344ae7521ced754c5bdbae6a not found: ID does not exist"
Jul 01 18:25:38.414966 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.414924    2577 scope.go:117] "RemoveContainer" containerID="f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f"
Jul 01 18:25:38.415202 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:25:38.415185    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f\": container with ID starting with f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f not found: ID does not exist" containerID="f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f"
Jul 01 18:25:38.415261 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.415214    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f"} err="failed to get container status \"f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f\": rpc error: code = NotFound desc = could not find container \"f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f\": container with ID starting with f453c42d5999ef08bbf89e428f9f58409533799425bc0419a792c8d46c88785f not found: ID does not exist"
Jul 01 18:25:38.445729 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.445696    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/625afa52-f752-4d0a-9a4b-318a66355400-proxy-tls\") pod \"625afa52-f752-4d0a-9a4b-318a66355400\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") "
Jul 01 18:25:38.445883 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.445743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-cabundle-cert\") pod \"625afa52-f752-4d0a-9a4b-318a66355400\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") "
Jul 01 18:25:38.445883 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.445786    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnbcb\" (UniqueName: \"kubernetes.io/projected/625afa52-f752-4d0a-9a4b-318a66355400-kube-api-access-cnbcb\") pod \"625afa52-f752-4d0a-9a4b-318a66355400\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") "
Jul 01 18:25:38.445883 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.445828    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-kube-rbac-proxy-sar-config\") pod \"625afa52-f752-4d0a-9a4b-318a66355400\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") "
Jul 01 18:25:38.446046 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.445969    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/625afa52-f752-4d0a-9a4b-318a66355400-kserve-provision-location\") pod \"625afa52-f752-4d0a-9a4b-318a66355400\" (UID: \"625afa52-f752-4d0a-9a4b-318a66355400\") "
Jul 01 18:25:38.446250 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.446211    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "625afa52-f752-4d0a-9a4b-318a66355400" (UID: "625afa52-f752-4d0a-9a4b-318a66355400"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:25:38.446377 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.446236    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "625afa52-f752-4d0a-9a4b-318a66355400" (UID: "625afa52-f752-4d0a-9a4b-318a66355400"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:25:38.446377 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.446233    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625afa52-f752-4d0a-9a4b-318a66355400-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "625afa52-f752-4d0a-9a4b-318a66355400" (UID: "625afa52-f752-4d0a-9a4b-318a66355400"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:25:38.447990 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.447970    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625afa52-f752-4d0a-9a4b-318a66355400-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "625afa52-f752-4d0a-9a4b-318a66355400" (UID: "625afa52-f752-4d0a-9a4b-318a66355400"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:25:38.448116 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.448096    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625afa52-f752-4d0a-9a4b-318a66355400-kube-api-access-cnbcb" (OuterVolumeSpecName: "kube-api-access-cnbcb") pod "625afa52-f752-4d0a-9a4b-318a66355400" (UID: "625afa52-f752-4d0a-9a4b-318a66355400"). InnerVolumeSpecName "kube-api-access-cnbcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:25:38.547650 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.547583    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/625afa52-f752-4d0a-9a4b-318a66355400-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:38.547650 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.547649    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/625afa52-f752-4d0a-9a4b-318a66355400-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:38.547860 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.547665    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:38.547860 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.547678    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnbcb\" (UniqueName: \"kubernetes.io/projected/625afa52-f752-4d0a-9a4b-318a66355400-kube-api-access-cnbcb\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:38.547860 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.547690    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/625afa52-f752-4d0a-9a4b-318a66355400-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:25:38.733970 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.733932    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"]
Jul 01 18:25:38.738802 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:38.738773    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb"]
Jul 01 18:25:39.020743 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.020649    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"]
Jul 01 18:25:39.021073 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.021055    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625afa52-f752-4d0a-9a4b-318a66355400" containerName="storage-initializer"
Jul 01 18:25:39.021159 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.021078    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="625afa52-f752-4d0a-9a4b-318a66355400" containerName="storage-initializer"
Jul 01 18:25:39.021217 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.021195    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="625afa52-f752-4d0a-9a4b-318a66355400" containerName="storage-initializer"
Jul 01 18:25:39.021217 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.021208    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="625afa52-f752-4d0a-9a4b-318a66355400" containerName="storage-initializer"
Jul 01 18:25:39.021323 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.021312    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625afa52-f752-4d0a-9a4b-318a66355400" containerName="storage-initializer"
Jul 01 18:25:39.021379 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.021324    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="625afa52-f752-4d0a-9a4b-318a66355400" containerName="storage-initializer"
Jul 01 18:25:39.025571 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.025547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.028389 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.028357    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:25:39.028389 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.028384    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-20421-predictor-serving-cert\""
Jul 01 18:25:39.028602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.028417    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-20421-kube-rbac-proxy-sar-config\""
Jul 01 18:25:39.028602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.028428    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 18:25:39.028602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.028415    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:25:39.028602 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.028536    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s5mjf\""
Jul 01 18:25:39.028817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.028704    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 18:25:39.036850 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.036827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"]
Jul 01 18:25:39.153723 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.153675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-cabundle-cert\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.153723 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.153724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-proxy-tls\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.153940 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.153812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kserve-provision-location\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.153940 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.153892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4mz\" (UniqueName: \"kubernetes.io/projected/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-api-access-7p4mz\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.153940 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.153933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.254899 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.254860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4mz\" (UniqueName: \"kubernetes.io/projected/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-api-access-7p4mz\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.255071 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.254921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.255071 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.254962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-cabundle-cert\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.255071 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.254989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-proxy-tls\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.255071 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.255052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kserve-provision-location\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.255489 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.255467    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kserve-provision-location\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.255720 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.255698    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.255782 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.255714    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-cabundle-cert\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.257665 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.257648    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-proxy-tls\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.262944 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.262919    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4mz\" (UniqueName: \"kubernetes.io/projected/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-api-access-7p4mz\") pod \"raw-sklearn-20421-predictor-58f54f4cf9-czsvq\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") " pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.336916 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.336816    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:39.473283 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.473247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"]
Jul 01 18:25:39.476472 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:25:39.476439    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ac156b_a0b7_4a6d_a3e1_9c26417aadcb.slice/crio-cd4f9f3623d0b50a8e82f6761fdee3a4fa9bdb411e15f764bde9fad6c687ce20 WatchSource:0}: Error finding container cd4f9f3623d0b50a8e82f6761fdee3a4fa9bdb411e15f764bde9fad6c687ce20: Status 404 returned error can't find the container with id cd4f9f3623d0b50a8e82f6761fdee3a4fa9bdb411e15f764bde9fad6c687ce20
Jul 01 18:25:39.869543 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:39.869506    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="625afa52-f752-4d0a-9a4b-318a66355400" path="/var/lib/kubelet/pods/625afa52-f752-4d0a-9a4b-318a66355400/volumes"
Jul 01 18:25:40.412200 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:40.412167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" event={"ID":"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb","Type":"ContainerStarted","Data":"1d2fbcaae62a910f9479db0708faa22c1be56456db1cf2c139341102e0b9a890"}
Jul 01 18:25:40.412570 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:40.412255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" event={"ID":"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb","Type":"ContainerStarted","Data":"cd4f9f3623d0b50a8e82f6761fdee3a4fa9bdb411e15f764bde9fad6c687ce20"}
Jul 01 18:25:43.428423 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:43.428391    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerID="1d2fbcaae62a910f9479db0708faa22c1be56456db1cf2c139341102e0b9a890" exitCode=0
Jul 01 18:25:43.428817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:43.428467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" event={"ID":"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb","Type":"ContainerDied","Data":"1d2fbcaae62a910f9479db0708faa22c1be56456db1cf2c139341102e0b9a890"}
Jul 01 18:25:44.435786 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:44.435752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" event={"ID":"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb","Type":"ContainerStarted","Data":"2f1f5445f9b0abc547db76c0c9a8633d62ed9b014fa29f56641927076d4b89c6"}
Jul 01 18:25:44.436217 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:44.435814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" event={"ID":"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb","Type":"ContainerStarted","Data":"7e97bd7bcbd1de3b10a21b6a09ff430edee8babc589abca7facbbf8a8d0bd940"}
Jul 01 18:25:44.436217 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:44.436078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:44.436217 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:44.436211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:44.437408 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:44.437381    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 18:25:44.453269 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:44.453217    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podStartSLOduration=1.8160537890000001 podStartE2EDuration="5.453199873s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.637146084s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:25:44.452693864 +0000 UTC m=+1175.014864500" watchObservedRunningTime="2026-07-01 18:25:44.453199873 +0000 UTC m=+1175.015370508"
Jul 01 18:25:45.441608 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:45.441573    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 18:25:50.446490 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:50.446463    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:25:50.446950 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:25:50.446925    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 18:26:00.447361 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:26:00.447318    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 18:26:09.896491 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:26:09.896467    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:26:09.901778 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:26:09.901754    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:26:10.447595 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:26:10.447559    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 18:26:20.446897 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:26:20.446855    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 18:26:30.447253 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:26:30.447208    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 18:26:40.447464 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:26:40.447423    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 18:26:50.447342 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:26:50.447303    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 18:27:00.447872 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:00.447793    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:27:09.193704 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:09.193661    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"]
Jul 01 18:27:09.194258 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:09.193992    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" containerID="cri-o://7e97bd7bcbd1de3b10a21b6a09ff430edee8babc589abca7facbbf8a8d0bd940" gracePeriod=30
Jul 01 18:27:09.194258 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:09.194039    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kube-rbac-proxy" containerID="cri-o://2f1f5445f9b0abc547db76c0c9a8633d62ed9b014fa29f56641927076d4b89c6" gracePeriod=30
Jul 01 18:27:09.805312 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:09.805276    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerID="2f1f5445f9b0abc547db76c0c9a8633d62ed9b014fa29f56641927076d4b89c6" exitCode=2
Jul 01 18:27:09.805491 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:09.805350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" event={"ID":"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb","Type":"ContainerDied","Data":"2f1f5445f9b0abc547db76c0c9a8633d62ed9b014fa29f56641927076d4b89c6"}
Jul 01 18:27:10.441817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:10.441781    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 01 18:27:10.447041 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:10.447017    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 18:27:13.823894 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.823863    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerID="7e97bd7bcbd1de3b10a21b6a09ff430edee8babc589abca7facbbf8a8d0bd940" exitCode=0
Jul 01 18:27:13.824240 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.823922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" event={"ID":"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb","Type":"ContainerDied","Data":"7e97bd7bcbd1de3b10a21b6a09ff430edee8babc589abca7facbbf8a8d0bd940"}
Jul 01 18:27:13.849599 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.849571    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:27:13.897057 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.897021    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-rbac-proxy-sar-config\") pod \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") "
Jul 01 18:27:13.897229 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.897128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-cabundle-cert\") pod \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") "
Jul 01 18:27:13.897229 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.897174    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-proxy-tls\") pod \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") "
Jul 01 18:27:13.897229 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.897201    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p4mz\" (UniqueName: \"kubernetes.io/projected/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-api-access-7p4mz\") pod \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") "
Jul 01 18:27:13.897392 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.897280    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" (UID: "f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:27:13.897504 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.897476    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" (UID: "f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:27:13.897668 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.897577    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:27:13.899908 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.899885    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" (UID: "f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:27:13.899994 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.899901    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-api-access-7p4mz" (OuterVolumeSpecName: "kube-api-access-7p4mz") pod "f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" (UID: "f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb"). InnerVolumeSpecName "kube-api-access-7p4mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:27:13.998365 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.998265    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kserve-provision-location\") pod \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\" (UID: \"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb\") "
Jul 01 18:27:13.998513 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.998443    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:27:13.998513 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.998457    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:27:13.998513 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.998465    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p4mz\" (UniqueName: \"kubernetes.io/projected/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kube-api-access-7p4mz\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:27:13.998664 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:13.998582    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" (UID: "f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:27:14.098935 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.098901    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:27:14.338784 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.338708    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"]
Jul 01 18:27:14.339069 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.339055    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kube-rbac-proxy"
Jul 01 18:27:14.339116 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.339072    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kube-rbac-proxy"
Jul 01 18:27:14.339116 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.339089    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="storage-initializer"
Jul 01 18:27:14.339116 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.339094    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="storage-initializer"
Jul 01 18:27:14.339116 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.339108    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container"
Jul 01 18:27:14.339116 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.339113    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container"
Jul 01 18:27:14.339271 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.339160    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kserve-container"
Jul 01 18:27:14.339271 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.339172    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" containerName="kube-rbac-proxy"
Jul 01 18:27:14.342541 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.342523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.344915 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.344896    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-10c8f-predictor-serving-cert\""
Jul 01 18:27:14.345002 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.344901    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-10c8f-kube-rbac-proxy-sar-config\""
Jul 01 18:27:14.351966 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.351939    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"]
Jul 01 18:27:14.402295 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.402260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-cabundle-cert\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.402459 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.402368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17063895-46d6-4356-b1f9-7ad387266e1a-proxy-tls\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.402459 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.402395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2nwq\" (UniqueName: \"kubernetes.io/projected/17063895-46d6-4356-b1f9-7ad387266e1a-kube-api-access-r2nwq\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.402459 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.402426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17063895-46d6-4356-b1f9-7ad387266e1a-kserve-provision-location\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.402459 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.402455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.502910 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.502874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17063895-46d6-4356-b1f9-7ad387266e1a-proxy-tls\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.502910 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.502914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2nwq\" (UniqueName: \"kubernetes.io/projected/17063895-46d6-4356-b1f9-7ad387266e1a-kube-api-access-r2nwq\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.503128 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.502959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17063895-46d6-4356-b1f9-7ad387266e1a-kserve-provision-location\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.503128 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.502986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.503128 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:27:14.503026    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-serving-cert: secret "raw-sklearn-runtime-10c8f-predictor-serving-cert" not found
Jul 01 18:27:14.503128 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.503043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-cabundle-cert\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.503128 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:27:14.503095    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17063895-46d6-4356-b1f9-7ad387266e1a-proxy-tls podName:17063895-46d6-4356-b1f9-7ad387266e1a nodeName:}" failed. No retries permitted until 2026-07-01 18:27:15.003071684 +0000 UTC m=+1265.565242315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/17063895-46d6-4356-b1f9-7ad387266e1a-proxy-tls") pod "raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" (UID: "17063895-46d6-4356-b1f9-7ad387266e1a") : secret "raw-sklearn-runtime-10c8f-predictor-serving-cert" not found
Jul 01 18:27:14.503442 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.503419    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17063895-46d6-4356-b1f9-7ad387266e1a-kserve-provision-location\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.503718 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.503703    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.503766 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.503715    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-cabundle-cert\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.511408 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.511385    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2nwq\" (UniqueName: \"kubernetes.io/projected/17063895-46d6-4356-b1f9-7ad387266e1a-kube-api-access-r2nwq\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:14.830538 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.830505    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq" event={"ID":"f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb","Type":"ContainerDied","Data":"cd4f9f3623d0b50a8e82f6761fdee3a4fa9bdb411e15f764bde9fad6c687ce20"}
Jul 01 18:27:14.830968 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.830581    2577 scope.go:117] "RemoveContainer" containerID="2f1f5445f9b0abc547db76c0c9a8633d62ed9b014fa29f56641927076d4b89c6"
Jul 01 18:27:14.830968 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.830515    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"
Jul 01 18:27:14.839163 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.839144    2577 scope.go:117] "RemoveContainer" containerID="7e97bd7bcbd1de3b10a21b6a09ff430edee8babc589abca7facbbf8a8d0bd940"
Jul 01 18:27:14.847189 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.847172    2577 scope.go:117] "RemoveContainer" containerID="1d2fbcaae62a910f9479db0708faa22c1be56456db1cf2c139341102e0b9a890"
Jul 01 18:27:14.854672 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.854604    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"]
Jul 01 18:27:14.858009 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:14.857985    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq"]
Jul 01 18:27:15.006223 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:15.006186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17063895-46d6-4356-b1f9-7ad387266e1a-proxy-tls\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:15.013740 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:15.013709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17063895-46d6-4356-b1f9-7ad387266e1a-proxy-tls\") pod \"raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:15.253335 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:15.253292    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:15.389707 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:15.389587    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"]
Jul 01 18:27:15.392405 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:27:15.392375    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17063895_46d6_4356_b1f9_7ad387266e1a.slice/crio-ad65562ebac2056640e780f5427fc92a47e258b6732225e8b147bd2adc755a9d WatchSource:0}: Error finding container ad65562ebac2056640e780f5427fc92a47e258b6732225e8b147bd2adc755a9d: Status 404 returned error can't find the container with id ad65562ebac2056640e780f5427fc92a47e258b6732225e8b147bd2adc755a9d
Jul 01 18:27:15.837695 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:15.837659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" event={"ID":"17063895-46d6-4356-b1f9-7ad387266e1a","Type":"ContainerStarted","Data":"b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047"}
Jul 01 18:27:15.838101 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:15.837738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" event={"ID":"17063895-46d6-4356-b1f9-7ad387266e1a","Type":"ContainerStarted","Data":"ad65562ebac2056640e780f5427fc92a47e258b6732225e8b147bd2adc755a9d"}
Jul 01 18:27:15.873072 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:15.873037    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb" path="/var/lib/kubelet/pods/f8ac156b-a0b7-4a6d-a3e1-9c26417aadcb/volumes"
Jul 01 18:27:19.857862 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:19.857826    2577 generic.go:356] "Generic (PLEG): container finished" podID="17063895-46d6-4356-b1f9-7ad387266e1a" containerID="b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047" exitCode=0
Jul 01 18:27:19.858324 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:19.857869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" event={"ID":"17063895-46d6-4356-b1f9-7ad387266e1a","Type":"ContainerDied","Data":"b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047"}
Jul 01 18:27:20.864815 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:20.864780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" event={"ID":"17063895-46d6-4356-b1f9-7ad387266e1a","Type":"ContainerStarted","Data":"08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb"}
Jul 01 18:27:20.865262 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:20.864841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" event={"ID":"17063895-46d6-4356-b1f9-7ad387266e1a","Type":"ContainerStarted","Data":"92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a"}
Jul 01 18:27:20.865262 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:20.865114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:20.865400 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:20.865278    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:20.866316 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:20.866289    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 18:27:20.884097 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:20.884052    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podStartSLOduration=3.22332259 podStartE2EDuration="6.884038221s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.660715631s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:27:20.88303782 +0000 UTC m=+1271.445208457" watchObservedRunningTime="2026-07-01 18:27:20.884038221 +0000 UTC m=+1271.446208857"
Jul 01 18:27:21.869656 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:21.869605    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 18:27:26.874168 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:26.874140    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:27:26.874665 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:26.874599    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 18:27:36.875174 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:36.875132    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 18:27:46.874839 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:46.874804    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 18:27:56.874626 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:27:56.874584    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 18:28:06.875356 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:06.875318    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 18:28:16.875249 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:16.875214    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 18:28:26.875450 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:26.875413    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 18:28:31.864929 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:31.864848    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 18:28:41.869137 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:41.869110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:28:44.437065 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:44.437031    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"]
Jul 01 18:28:44.437511 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:44.437250    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container" containerID="cri-o://92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a" gracePeriod=30
Jul 01 18:28:44.437511 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:44.437299    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kube-rbac-proxy" containerID="cri-o://08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb" gracePeriod=30
Jul 01 18:28:45.228405 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:45.228375    2577 generic.go:356] "Generic (PLEG): container finished" podID="17063895-46d6-4356-b1f9-7ad387266e1a" containerID="08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb" exitCode=2
Jul 01 18:28:45.228405 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:45.228419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" event={"ID":"17063895-46d6-4356-b1f9-7ad387266e1a","Type":"ContainerDied","Data":"08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb"}
Jul 01 18:28:46.870712 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:46.870671    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 01 18:28:49.087244 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.087219    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:28:49.231299 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.231195    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2nwq\" (UniqueName: \"kubernetes.io/projected/17063895-46d6-4356-b1f9-7ad387266e1a-kube-api-access-r2nwq\") pod \"17063895-46d6-4356-b1f9-7ad387266e1a\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") "
Jul 01 18:28:49.231299 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.231238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-kube-rbac-proxy-sar-config\") pod \"17063895-46d6-4356-b1f9-7ad387266e1a\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") "
Jul 01 18:28:49.231299 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.231285    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17063895-46d6-4356-b1f9-7ad387266e1a-kserve-provision-location\") pod \"17063895-46d6-4356-b1f9-7ad387266e1a\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") "
Jul 01 18:28:49.231599 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.231330    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17063895-46d6-4356-b1f9-7ad387266e1a-proxy-tls\") pod \"17063895-46d6-4356-b1f9-7ad387266e1a\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") "
Jul 01 18:28:49.231599 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.231484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-cabundle-cert\") pod \"17063895-46d6-4356-b1f9-7ad387266e1a\" (UID: \"17063895-46d6-4356-b1f9-7ad387266e1a\") "
Jul 01 18:28:49.231758 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.231694    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17063895-46d6-4356-b1f9-7ad387266e1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17063895-46d6-4356-b1f9-7ad387266e1a" (UID: "17063895-46d6-4356-b1f9-7ad387266e1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:28:49.231758 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.231707    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "17063895-46d6-4356-b1f9-7ad387266e1a" (UID: "17063895-46d6-4356-b1f9-7ad387266e1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:28:49.231908 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.231882    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17063895-46d6-4356-b1f9-7ad387266e1a" (UID: "17063895-46d6-4356-b1f9-7ad387266e1a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:28:49.231962 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.231898    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:28:49.231962 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.231939    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17063895-46d6-4356-b1f9-7ad387266e1a-kserve-provision-location\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:28:49.233608 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.233579    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17063895-46d6-4356-b1f9-7ad387266e1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17063895-46d6-4356-b1f9-7ad387266e1a" (UID: "17063895-46d6-4356-b1f9-7ad387266e1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:28:49.233608 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.233590    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17063895-46d6-4356-b1f9-7ad387266e1a-kube-api-access-r2nwq" (OuterVolumeSpecName: "kube-api-access-r2nwq") pod "17063895-46d6-4356-b1f9-7ad387266e1a" (UID: "17063895-46d6-4356-b1f9-7ad387266e1a"). InnerVolumeSpecName "kube-api-access-r2nwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:28:49.252102 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.252072    2577 generic.go:356] "Generic (PLEG): container finished" podID="17063895-46d6-4356-b1f9-7ad387266e1a" containerID="92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a" exitCode=0
Jul 01 18:28:49.252251 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.252127    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"
Jul 01 18:28:49.252251 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.252158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" event={"ID":"17063895-46d6-4356-b1f9-7ad387266e1a","Type":"ContainerDied","Data":"92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a"}
Jul 01 18:28:49.252251 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.252220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6" event={"ID":"17063895-46d6-4356-b1f9-7ad387266e1a","Type":"ContainerDied","Data":"ad65562ebac2056640e780f5427fc92a47e258b6732225e8b147bd2adc755a9d"}
Jul 01 18:28:49.252251 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.252249    2577 scope.go:117] "RemoveContainer" containerID="08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb"
Jul 01 18:28:49.261393 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.261369    2577 scope.go:117] "RemoveContainer" containerID="92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a"
Jul 01 18:28:49.269119 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.269098    2577 scope.go:117] "RemoveContainer" containerID="b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047"
Jul 01 18:28:49.276282 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.276259    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"]
Jul 01 18:28:49.276599 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.276586    2577 scope.go:117] "RemoveContainer" containerID="08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb"
Jul 01 18:28:49.276913 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:28:49.276888    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb\": container with ID starting with 08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb not found: ID does not exist" containerID="08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb"
Jul 01 18:28:49.277004 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.276926    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb"} err="failed to get container status \"08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb\": rpc error: code = NotFound desc = could not find container \"08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb\": container with ID starting with 08d1a41c40c1342ea43f414071c424afbfc0a9efa7a9dd5621cc1497ba5da3eb not found: ID does not exist"
Jul 01 18:28:49.277004 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.276945    2577 scope.go:117] "RemoveContainer" containerID="92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a"
Jul 01 18:28:49.277207 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:28:49.277190    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a\": container with ID starting with 92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a not found: ID does not exist" containerID="92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a"
Jul 01 18:28:49.277271 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.277217    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a"} err="failed to get container status \"92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a\": rpc error: code = NotFound desc = could not find container \"92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a\": container with ID starting with 92131862ef028c24478f203d228f7155ed83be3d3871297a286a0fdf686aeb3a not found: ID does not exist"
Jul 01 18:28:49.277271 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.277239    2577 scope.go:117] "RemoveContainer" containerID="b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047"
Jul 01 18:28:49.277486 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:28:49.277471    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047\": container with ID starting with b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047 not found: ID does not exist" containerID="b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047"
Jul 01 18:28:49.277565 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.277491    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047"} err="failed to get container status \"b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047\": rpc error: code = NotFound desc = could not find container \"b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047\": container with ID starting with b365ca48ca71fd3bd14e01de063e2106e15073a20a41123e00ca61fa03a33047 not found: ID does not exist"
Jul 01 18:28:49.280966 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.280945    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6"]
Jul 01 18:28:49.332714 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.332683    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17063895-46d6-4356-b1f9-7ad387266e1a-proxy-tls\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:28:49.332714 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.332708    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17063895-46d6-4356-b1f9-7ad387266e1a-cabundle-cert\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:28:49.332714 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.332719    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2nwq\" (UniqueName: \"kubernetes.io/projected/17063895-46d6-4356-b1f9-7ad387266e1a-kube-api-access-r2nwq\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:28:49.869403 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:49.869368    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" path="/var/lib/kubelet/pods/17063895-46d6-4356-b1f9-7ad387266e1a/volumes"
Jul 01 18:28:51.007503 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.007468    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9svxw/must-gather-5tqvk"]
Jul 01 18:28:51.007896 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.007820    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="storage-initializer"
Jul 01 18:28:51.007896 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.007830    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="storage-initializer"
Jul 01 18:28:51.007896 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.007847    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container"
Jul 01 18:28:51.007896 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.007853    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container"
Jul 01 18:28:51.007896 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.007865    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kube-rbac-proxy"
Jul 01 18:28:51.007896 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.007871    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kube-rbac-proxy"
Jul 01 18:28:51.008096 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.007916    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kserve-container"
Jul 01 18:28:51.008096 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.007924    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17063895-46d6-4356-b1f9-7ad387266e1a" containerName="kube-rbac-proxy"
Jul 01 18:28:51.012270 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.012253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:28:51.014905 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.014883    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9svxw\"/\"openshift-service-ca.crt\""
Jul 01 18:28:51.015025 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.014913    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9svxw\"/\"default-dockercfg-966r9\""
Jul 01 18:28:51.015025 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.014981    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9svxw\"/\"kube-root-ca.crt\""
Jul 01 18:28:51.019744 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.019717    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9svxw/must-gather-5tqvk"]
Jul 01 18:28:51.148838 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.148794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98sqw\" (UniqueName: \"kubernetes.io/projected/31821e58-2692-497f-a79d-d3bdbd54aa04-kube-api-access-98sqw\") pod \"must-gather-5tqvk\" (UID: \"31821e58-2692-497f-a79d-d3bdbd54aa04\") " pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:28:51.148838 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.148844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31821e58-2692-497f-a79d-d3bdbd54aa04-must-gather-output\") pod \"must-gather-5tqvk\" (UID: \"31821e58-2692-497f-a79d-d3bdbd54aa04\") " pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:28:51.249951 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.249919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98sqw\" (UniqueName: \"kubernetes.io/projected/31821e58-2692-497f-a79d-d3bdbd54aa04-kube-api-access-98sqw\") pod \"must-gather-5tqvk\" (UID: \"31821e58-2692-497f-a79d-d3bdbd54aa04\") " pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:28:51.250071 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.249958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31821e58-2692-497f-a79d-d3bdbd54aa04-must-gather-output\") pod \"must-gather-5tqvk\" (UID: \"31821e58-2692-497f-a79d-d3bdbd54aa04\") " pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:28:51.250257 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.250241    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31821e58-2692-497f-a79d-d3bdbd54aa04-must-gather-output\") pod \"must-gather-5tqvk\" (UID: \"31821e58-2692-497f-a79d-d3bdbd54aa04\") " pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:28:51.258775 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.258709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98sqw\" (UniqueName: \"kubernetes.io/projected/31821e58-2692-497f-a79d-d3bdbd54aa04-kube-api-access-98sqw\") pod \"must-gather-5tqvk\" (UID: \"31821e58-2692-497f-a79d-d3bdbd54aa04\") " pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:28:51.332657 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.332605    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:28:51.465912 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.465886    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9svxw/must-gather-5tqvk"]
Jul 01 18:28:51.468313 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:28:51.468288    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31821e58_2692_497f_a79d_d3bdbd54aa04.slice/crio-bd9a3ee3dd74cd06b46893821b6c96f26f05089d5391f0d465077d28d821cc5f WatchSource:0}: Error finding container bd9a3ee3dd74cd06b46893821b6c96f26f05089d5391f0d465077d28d821cc5f: Status 404 returned error can't find the container with id bd9a3ee3dd74cd06b46893821b6c96f26f05089d5391f0d465077d28d821cc5f
Jul 01 18:28:51.470181 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:51.470165    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:28:52.269095 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:52.268977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9svxw/must-gather-5tqvk" event={"ID":"31821e58-2692-497f-a79d-d3bdbd54aa04","Type":"ContainerStarted","Data":"bd9a3ee3dd74cd06b46893821b6c96f26f05089d5391f0d465077d28d821cc5f"}
Jul 01 18:28:58.300307 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:58.300270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9svxw/must-gather-5tqvk" event={"ID":"31821e58-2692-497f-a79d-d3bdbd54aa04","Type":"ContainerStarted","Data":"f05e31ee20ece1879d263e7f88fa4ec2d96f00834759d93ca432568630a14766"}
Jul 01 18:28:58.300307 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:58.300316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9svxw/must-gather-5tqvk" event={"ID":"31821e58-2692-497f-a79d-d3bdbd54aa04","Type":"ContainerStarted","Data":"87fd0ca6f9892f15d66a137bad040d9269f8887c02be42485fc3621e8e208faf"}
Jul 01 18:28:58.316584 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:28:58.316535    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9svxw/must-gather-5tqvk" podStartSLOduration=2.231729229 podStartE2EDuration="8.316522449s" totalImagesPullingTime="6.08479322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:28:58.315710508 +0000 UTC m=+1368.877881145" watchObservedRunningTime="2026-07-01 18:28:58.316522449 +0000 UTC m=+1368.878693084"
Jul 01 18:29:19.394547 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:19.394513    2577 generic.go:356] "Generic (PLEG): container finished" podID="31821e58-2692-497f-a79d-d3bdbd54aa04" containerID="87fd0ca6f9892f15d66a137bad040d9269f8887c02be42485fc3621e8e208faf" exitCode=0
Jul 01 18:29:19.395007 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:19.394584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9svxw/must-gather-5tqvk" event={"ID":"31821e58-2692-497f-a79d-d3bdbd54aa04","Type":"ContainerDied","Data":"87fd0ca6f9892f15d66a137bad040d9269f8887c02be42485fc3621e8e208faf"}
Jul 01 18:29:19.395007 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:19.394936    2577 scope.go:117] "RemoveContainer" containerID="87fd0ca6f9892f15d66a137bad040d9269f8887c02be42485fc3621e8e208faf"
Jul 01 18:29:19.797913 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:19.797834    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9svxw_must-gather-5tqvk_31821e58-2692-497f-a79d-d3bdbd54aa04/gather/0.log"
Jul 01 18:29:23.374210 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:23.374181    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dkxdw_8e574eeb-3e4c-4e1c-9e54-9b1bf45f2787/global-pull-secret-syncer/0.log"
Jul 01 18:29:23.573972 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:23.573949    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-phk7j_1592b6ff-d1f0-4f69-b575-659063252392/konnectivity-agent/0.log"
Jul 01 18:29:23.593747 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:23.593725    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-145.ec2.internal_745eb6e171aca622db7fea2abbb34988/haproxy/0.log"
Jul 01 18:29:25.221953 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.221911    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-9svxw/must-gather-5tqvk"]
Jul 01 18:29:25.222398 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.222139    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-9svxw/must-gather-5tqvk" podUID="31821e58-2692-497f-a79d-d3bdbd54aa04" containerName="copy" containerID="cri-o://f05e31ee20ece1879d263e7f88fa4ec2d96f00834759d93ca432568630a14766" gracePeriod=2
Jul 01 18:29:25.226603 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.225985    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-9svxw/must-gather-5tqvk"]
Jul 01 18:29:25.420672 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.420646    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9svxw_must-gather-5tqvk_31821e58-2692-497f-a79d-d3bdbd54aa04/copy/0.log"
Jul 01 18:29:25.421049 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.421024    2577 generic.go:356] "Generic (PLEG): container finished" podID="31821e58-2692-497f-a79d-d3bdbd54aa04" containerID="f05e31ee20ece1879d263e7f88fa4ec2d96f00834759d93ca432568630a14766" exitCode=143
Jul 01 18:29:25.462365 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.462334    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9svxw_must-gather-5tqvk_31821e58-2692-497f-a79d-d3bdbd54aa04/copy/0.log"
Jul 01 18:29:25.463799 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.463780    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:29:25.466114 ip-10-0-132-145 kubenswrapper[2577]: E0701 18:29:25.466089    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5tqvk\" is forbidden: User \"system:node:ip-10-0-132-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9svxw\": no relationship found between node 'ip-10-0-132-145.ec2.internal' and this object" podUID="31821e58-2692-497f-a79d-d3bdbd54aa04" pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:29:25.573646 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.573546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98sqw\" (UniqueName: \"kubernetes.io/projected/31821e58-2692-497f-a79d-d3bdbd54aa04-kube-api-access-98sqw\") pod \"31821e58-2692-497f-a79d-d3bdbd54aa04\" (UID: \"31821e58-2692-497f-a79d-d3bdbd54aa04\") "
Jul 01 18:29:25.573646 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.573631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31821e58-2692-497f-a79d-d3bdbd54aa04-must-gather-output\") pod \"31821e58-2692-497f-a79d-d3bdbd54aa04\" (UID: \"31821e58-2692-497f-a79d-d3bdbd54aa04\") "
Jul 01 18:29:25.574945 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.574911    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31821e58-2692-497f-a79d-d3bdbd54aa04-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "31821e58-2692-497f-a79d-d3bdbd54aa04" (UID: "31821e58-2692-497f-a79d-d3bdbd54aa04"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:29:25.575911 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.575889    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31821e58-2692-497f-a79d-d3bdbd54aa04-kube-api-access-98sqw" (OuterVolumeSpecName: "kube-api-access-98sqw") pod "31821e58-2692-497f-a79d-d3bdbd54aa04" (UID: "31821e58-2692-497f-a79d-d3bdbd54aa04"). InnerVolumeSpecName "kube-api-access-98sqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:29:25.674767 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.674728    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98sqw\" (UniqueName: \"kubernetes.io/projected/31821e58-2692-497f-a79d-d3bdbd54aa04-kube-api-access-98sqw\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:29:25.674767 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.674764    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31821e58-2692-497f-a79d-d3bdbd54aa04-must-gather-output\") on node \"ip-10-0-132-145.ec2.internal\" DevicePath \"\""
Jul 01 18:29:25.868947 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:25.868915    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31821e58-2692-497f-a79d-d3bdbd54aa04" path="/var/lib/kubelet/pods/31821e58-2692-497f-a79d-d3bdbd54aa04/volumes"
Jul 01 18:29:26.432228 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:26.432198    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9svxw_must-gather-5tqvk_31821e58-2692-497f-a79d-d3bdbd54aa04/copy/0.log"
Jul 01 18:29:26.432708 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:26.432593    2577 scope.go:117] "RemoveContainer" containerID="f05e31ee20ece1879d263e7f88fa4ec2d96f00834759d93ca432568630a14766"
Jul 01 18:29:26.432708 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:26.432637    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9svxw/must-gather-5tqvk"
Jul 01 18:29:26.440429 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:26.440412    2577 scope.go:117] "RemoveContainer" containerID="87fd0ca6f9892f15d66a137bad040d9269f8887c02be42485fc3621e8e208faf"
Jul 01 18:29:28.100107 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.100078    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ddea599-b77d-4f0e-9bfc-9603605d1141/alertmanager/0.log"
Jul 01 18:29:28.132544 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.132518    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ddea599-b77d-4f0e-9bfc-9603605d1141/config-reloader/0.log"
Jul 01 18:29:28.162348 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.162315    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ddea599-b77d-4f0e-9bfc-9603605d1141/kube-rbac-proxy-web/0.log"
Jul 01 18:29:28.191803 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.191780    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ddea599-b77d-4f0e-9bfc-9603605d1141/kube-rbac-proxy/0.log"
Jul 01 18:29:28.214557 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.214536    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ddea599-b77d-4f0e-9bfc-9603605d1141/kube-rbac-proxy-metric/0.log"
Jul 01 18:29:28.237466 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.237442    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ddea599-b77d-4f0e-9bfc-9603605d1141/prom-label-proxy/0.log"
Jul 01 18:29:28.263052 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.263027    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5ddea599-b77d-4f0e-9bfc-9603605d1141/init-config-reloader/0.log"
Jul 01 18:29:28.336592 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.336563    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-zmrjk_728eca3a-b50a-46ca-ac9e-62f5fcec1839/kube-state-metrics/0.log"
Jul 01 18:29:28.367981 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.367910    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-zmrjk_728eca3a-b50a-46ca-ac9e-62f5fcec1839/kube-rbac-proxy-main/0.log"
Jul 01 18:29:28.394141 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.394114    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-zmrjk_728eca3a-b50a-46ca-ac9e-62f5fcec1839/kube-rbac-proxy-self/0.log"
Jul 01 18:29:28.427659 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.427601    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9b9977845-hjnlw_fb821ca4-9c9e-4fc4-903e-3c932c30bd87/metrics-server/0.log"
Jul 01 18:29:28.461964 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.461936    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-cd8656b8-hm9jk_910e51e8-215b-40ab-807d-cf3df2229643/monitoring-plugin/0.log"
Jul 01 18:29:28.509124 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.509095    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hkt6w_e924e627-b425-4b38-8991-7954dc89ecf8/node-exporter/0.log"
Jul 01 18:29:28.544584 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.544559    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hkt6w_e924e627-b425-4b38-8991-7954dc89ecf8/kube-rbac-proxy/0.log"
Jul 01 18:29:28.568658 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.568636    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hkt6w_e924e627-b425-4b38-8991-7954dc89ecf8/init-textfile/0.log"
Jul 01 18:29:28.751860 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.751786    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-w92fp_9e9d4acf-3824-409a-806e-b297e5daeaf8/kube-rbac-proxy-main/0.log"
Jul 01 18:29:28.790714 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.790682    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-w92fp_9e9d4acf-3824-409a-806e-b297e5daeaf8/kube-rbac-proxy-self/0.log"
Jul 01 18:29:28.826178 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.826151    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-w92fp_9e9d4acf-3824-409a-806e-b297e5daeaf8/openshift-state-metrics/0.log"
Jul 01 18:29:28.878365 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.878328    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094/prometheus/0.log"
Jul 01 18:29:28.916315 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.916291    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094/config-reloader/0.log"
Jul 01 18:29:28.946326 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.946305    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094/thanos-sidecar/0.log"
Jul 01 18:29:28.971110 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:28.971082    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094/kube-rbac-proxy-web/0.log"
Jul 01 18:29:29.003660 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.003571    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094/kube-rbac-proxy/0.log"
Jul 01 18:29:29.034659 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.034637    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094/kube-rbac-proxy-thanos/0.log"
Jul 01 18:29:29.066878 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.066852    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9b3cb9ec-a6b0-4f0f-a4eb-9064a37f3094/init-config-reloader/0.log"
Jul 01 18:29:29.105633 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.105591    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-cbxqd_7bcd46d5-340a-4a86-8766-1836a6b31198/prometheus-operator/0.log"
Jul 01 18:29:29.131052 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.131035    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-cbxqd_7bcd46d5-340a-4a86-8766-1836a6b31198/kube-rbac-proxy/0.log"
Jul 01 18:29:29.184699 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.184676    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d747cb99-5tb2q_cda60a1f-0aa6-43dd-b53b-28356df77b66/telemeter-client/0.log"
Jul 01 18:29:29.207417 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.207387    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d747cb99-5tb2q_cda60a1f-0aa6-43dd-b53b-28356df77b66/reload/0.log"
Jul 01 18:29:29.228624 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.228602    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d747cb99-5tb2q_cda60a1f-0aa6-43dd-b53b-28356df77b66/kube-rbac-proxy/0.log"
Jul 01 18:29:29.262672 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.262587    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675f8b559-zb88l_a0e787a9-dec5-4d9b-b0f5-cfb65785e82a/thanos-query/0.log"
Jul 01 18:29:29.284848 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.284828    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675f8b559-zb88l_a0e787a9-dec5-4d9b-b0f5-cfb65785e82a/kube-rbac-proxy-web/0.log"
Jul 01 18:29:29.313372 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.313351    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675f8b559-zb88l_a0e787a9-dec5-4d9b-b0f5-cfb65785e82a/kube-rbac-proxy/0.log"
Jul 01 18:29:29.339817 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.339800    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675f8b559-zb88l_a0e787a9-dec5-4d9b-b0f5-cfb65785e82a/prom-label-proxy/0.log"
Jul 01 18:29:29.364364 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.364343    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675f8b559-zb88l_a0e787a9-dec5-4d9b-b0f5-cfb65785e82a/kube-rbac-proxy-rules/0.log"
Jul 01 18:29:29.387890 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:29.387870    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675f8b559-zb88l_a0e787a9-dec5-4d9b-b0f5-cfb65785e82a/kube-rbac-proxy-metrics/0.log"
Jul 01 18:29:31.367729 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.367692    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"]
Jul 01 18:29:31.368097 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.368020    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31821e58-2692-497f-a79d-d3bdbd54aa04" containerName="copy"
Jul 01 18:29:31.368097 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.368031    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31821e58-2692-497f-a79d-d3bdbd54aa04" containerName="copy"
Jul 01 18:29:31.368097 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.368047    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31821e58-2692-497f-a79d-d3bdbd54aa04" containerName="gather"
Jul 01 18:29:31.368097 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.368052    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31821e58-2692-497f-a79d-d3bdbd54aa04" containerName="gather"
Jul 01 18:29:31.368273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.368110    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31821e58-2692-497f-a79d-d3bdbd54aa04" containerName="copy"
Jul 01 18:29:31.368273 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.368122    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31821e58-2692-497f-a79d-d3bdbd54aa04" containerName="gather"
Jul 01 18:29:31.372056 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.372033    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.374681 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.374660    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w9pcn\"/\"openshift-service-ca.crt\""
Jul 01 18:29:31.374795 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.374662    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w9pcn\"/\"kube-root-ca.crt\""
Jul 01 18:29:31.375582 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.375568    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w9pcn\"/\"default-dockercfg-6d9kw\""
Jul 01 18:29:31.379291 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.379268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"]
Jul 01 18:29:31.420374 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.420339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-podres\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.420548 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.420385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7k5l\" (UniqueName: \"kubernetes.io/projected/0d507d7b-806e-4efc-90b3-f6bab60683b4-kube-api-access-f7k5l\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.420548 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.420441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-proc\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.420548 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.420513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-sys\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.420687 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.420558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-lib-modules\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.521824 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.521783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-podres\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.522010 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.521833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7k5l\" (UniqueName: \"kubernetes.io/projected/0d507d7b-806e-4efc-90b3-f6bab60683b4-kube-api-access-f7k5l\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.522010 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.521883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-proc\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.522010 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.521901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-sys\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.522010 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.521942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-lib-modules\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.522010 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.521958    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-podres\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.522202 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.522010    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-sys\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.522202 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.522004    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-proc\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.522202 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.522075    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d507d7b-806e-4efc-90b3-f6bab60683b4-lib-modules\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.530391 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.530366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7k5l\" (UniqueName: \"kubernetes.io/projected/0d507d7b-806e-4efc-90b3-f6bab60683b4-kube-api-access-f7k5l\") pod \"perf-node-gather-daemonset-zxb6v\" (UID: \"0d507d7b-806e-4efc-90b3-f6bab60683b4\") " pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.683757 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.683677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:31.812212 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:31.812185    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"]
Jul 01 18:29:31.815032 ip-10-0-132-145 kubenswrapper[2577]: W0701 18:29:31.815001    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0d507d7b_806e_4efc_90b3_f6bab60683b4.slice/crio-ffd2715e35fe94b839f10faa66bd791a94c844efeb6d3970597f3b8d447fad12 WatchSource:0}: Error finding container ffd2715e35fe94b839f10faa66bd791a94c844efeb6d3970597f3b8d447fad12: Status 404 returned error can't find the container with id ffd2715e35fe94b839f10faa66bd791a94c844efeb6d3970597f3b8d447fad12
Jul 01 18:29:32.463400 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:32.463368    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v" event={"ID":"0d507d7b-806e-4efc-90b3-f6bab60683b4","Type":"ContainerStarted","Data":"fb70c92ec6e4993cc72c1b472a52edd09fefabb0097abc4113c7760a71b52719"}
Jul 01 18:29:32.463808 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:32.463422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v" event={"ID":"0d507d7b-806e-4efc-90b3-f6bab60683b4","Type":"ContainerStarted","Data":"ffd2715e35fe94b839f10faa66bd791a94c844efeb6d3970597f3b8d447fad12"}
Jul 01 18:29:32.463808 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:32.463454    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:32.481250 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:32.481199    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v" podStartSLOduration=1.4811827229999999 podStartE2EDuration="1.481182723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:29:32.479606398 +0000 UTC m=+1403.041777036" watchObservedRunningTime="2026-07-01 18:29:32.481182723 +0000 UTC m=+1403.043353359"
Jul 01 18:29:32.779244 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:32.779165    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rx2b6_aafab7ef-f502-4354-a7aa-d42d504684dd/dns/0.log"
Jul 01 18:29:32.802238 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:32.802214    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rx2b6_aafab7ef-f502-4354-a7aa-d42d504684dd/kube-rbac-proxy/0.log"
Jul 01 18:29:32.932481 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:32.932453    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xlgql_8f91615a-7c42-46c7-b117-fc873db099a8/dns-node-resolver/0.log"
Jul 01 18:29:33.402539 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:33.402501    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hssbk_1fc44217-ec85-4856-bd89-c15861990076/node-ca/0.log"
Jul 01 18:29:34.583990 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:34.583960    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7mtzl_89bd1a5c-5b4a-4212-b175-2157b3550774/serve-healthcheck-canary/0.log"
Jul 01 18:29:35.088395 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:35.088367    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fjkjl_98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34/kube-rbac-proxy/0.log"
Jul 01 18:29:35.109093 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:35.109067    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fjkjl_98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34/exporter/0.log"
Jul 01 18:29:35.135151 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:35.135119    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fjkjl_98f2c78e-7b2c-4ab3-8c81-5bfb73ae4d34/extractor/0.log"
Jul 01 18:29:37.259802 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:37.259772    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-jwws6_22235a60-9ad3-4ad5-8a29-2377aad43187/server/0.log"
Jul 01 18:29:37.435843 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:37.435810    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5kb5f_e1f01ea7-3fa7-4d90-b2c9-091c37b12550/s3-init/0.log"
Jul 01 18:29:37.451052 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:37.451032    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5kb5f_e1f01ea7-3fa7-4d90-b2c9-091c37b12550/download-hf-model/0.log"
Jul 01 18:29:38.477165 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:38.477137    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w9pcn/perf-node-gather-daemonset-zxb6v"
Jul 01 18:29:43.178374 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:43.178332    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vnxkj_79d3fc42-2365-420d-9acd-27da9298db0c/kube-multus-additional-cni-plugins/0.log"
Jul 01 18:29:43.202737 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:43.202704    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vnxkj_79d3fc42-2365-420d-9acd-27da9298db0c/egress-router-binary-copy/0.log"
Jul 01 18:29:43.229003 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:43.228975    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vnxkj_79d3fc42-2365-420d-9acd-27da9298db0c/cni-plugins/0.log"
Jul 01 18:29:43.251603 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:43.251578    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vnxkj_79d3fc42-2365-420d-9acd-27da9298db0c/bond-cni-plugin/0.log"
Jul 01 18:29:43.272822 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:43.272799    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vnxkj_79d3fc42-2365-420d-9acd-27da9298db0c/routeoverride-cni/0.log"
Jul 01 18:29:43.295808 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:43.295782    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vnxkj_79d3fc42-2365-420d-9acd-27da9298db0c/whereabouts-cni-bincopy/0.log"
Jul 01 18:29:43.317633 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:43.317595    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vnxkj_79d3fc42-2365-420d-9acd-27da9298db0c/whereabouts-cni/0.log"
Jul 01 18:29:43.348331 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:43.348303    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fw7vg_70bf894a-4512-411a-9b24-0490ec0ec047/kube-multus/0.log"
Jul 01 18:29:43.529169 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:43.529089    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qkfdw_51b74b41-8fc3-4992-9a5a-f3f01dfde169/network-metrics-daemon/0.log"
Jul 01 18:29:43.549040 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:43.549010    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qkfdw_51b74b41-8fc3-4992-9a5a-f3f01dfde169/kube-rbac-proxy/0.log"
Jul 01 18:29:44.321378 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:44.321347    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-controller/0.log"
Jul 01 18:29:44.340774 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:44.340745    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/0.log"
Jul 01 18:29:44.340922 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:44.340782    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovn-acl-logging/1.log"
Jul 01 18:29:44.356820 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:44.356796    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/kube-rbac-proxy-node/0.log"
Jul 01 18:29:44.378091 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:44.378055    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 18:29:44.401634 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:44.401588    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/northd/0.log"
Jul 01 18:29:44.424476 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:44.424453    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/nbdb/0.log"
Jul 01 18:29:44.451028 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:44.451003    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/sbdb/0.log"
Jul 01 18:29:44.499774 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:44.499742    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44jxb_b1aae139-cdf7-4c0d-b289-13a85e9000d9/ovnkube-controller/0.log"
Jul 01 18:29:45.978297 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:45.978272    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-stnmw_51bddbdc-6a46-4d8c-a6ea-f92883c6189a/network-check-target-container/0.log"
Jul 01 18:29:46.823307 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:46.823284    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-mmgrz_4fb2f666-408e-4c90-83fb-7386c9dca775/iptables-alerter/0.log"
Jul 01 18:29:47.565586 ip-10-0-132-145 kubenswrapper[2577]: I0701 18:29:47.565555    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rc4mf_3e73643a-f248-43d2-9433-f8fc9fa89449/tuned/0.log"