Jul 01 14:39:10.061175 ip-10-0-134-20 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 14:39:10.443892 ip-10-0-134-20 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 14:39:10.443892 ip-10-0-134-20 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 14:39:10.443892 ip-10-0-134-20 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 14:39:10.443892 ip-10-0-134-20 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 14:39:10.443892 ip-10-0-134-20 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 14:39:10.444800 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.444724    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 14:39:10.448745 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448731    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:39:10.448745 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448746    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448750    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448753    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448756    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448759    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448762    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448764    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448767    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448770    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448772    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448775    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448777    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448781    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448784    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448787    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448789    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448793    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448795    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448798    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448800    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:39:10.448809 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448803    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448805    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448808    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448811    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448814    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448817    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448819    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448823    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448827    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448830    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448833    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448836    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448839    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448841    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448844    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448847    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448849    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448852    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448854    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448857    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:39:10.449303 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448859    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448861    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448864    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448866    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448868    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448871    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448873    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448876    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448878    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448881    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448884    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448886    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448888    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448891    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448894    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448908    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448911    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448915    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448920    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:39:10.449847 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448923    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448927    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448930    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448933    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448936    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448938    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448941    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448943    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448946    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448948    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448951    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448953    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448955    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448958    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448960    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448962    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448965    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448967    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448971    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448973    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:39:10.450572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448976    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448979    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448981    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448990    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448993    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448995    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.448998    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449001    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449003    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449006    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449009    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449011    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449014    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449016    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449019    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449021    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449024    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449027    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.449029    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450817    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:39:10.451158 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450837    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450847    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450852    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450857    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450862    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450866    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450870    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450875    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450880    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.450884    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451052    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451062    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451066    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451069    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451072    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451075    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451078    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451081    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451084    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451087    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:39:10.451625 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451089    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451096    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451103    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451108    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451113    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451116    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451119    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451122    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451125    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451130    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451132    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451136    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451140    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451142    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451145    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451148    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451151    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451154    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451165    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:39:10.452172 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451168    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451171    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451174    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451176    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451179    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451182    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451184    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451187    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451190    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451192    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451195    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451200    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451203    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451205    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451208    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451211    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451214    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451218    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451220    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451223    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:39:10.452631 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451226    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451228    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451233    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451237    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451240    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451243    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451245    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451248    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451251    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451254    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451256    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451259    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451261    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451264    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451266    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451269    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451271    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451274    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451277    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:39:10.453122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451280    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451282    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451285    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451287    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451291    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451294    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451297    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451299    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451302    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451304    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451307    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451312    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451314    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451317    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451319    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451323    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451326    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451329    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451331    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451334    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:39:10.453572 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451717    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451722    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451725    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451728    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451731    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451733    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451736    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451739    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451741    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451744    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451746    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451749    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451752    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451755    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451757    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451760    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451764    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451767    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451769    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:39:10.454064 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451772    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451774    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451777    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451780    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451782    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451785    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451788    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451790    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451793    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451796    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451798    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451801    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451804    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451806    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451809    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451811    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451813    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451816    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451818    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451821    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:39:10.454528 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451823    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451826    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451828    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451830    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451833    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451835    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451838    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451840    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451843    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451846    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451849    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451851    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451854    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451858    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451861    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451864    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451867    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451870    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451872    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:39:10.455034 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451875    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451878    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451881    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451883    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451886    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451889    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451891    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451894    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451910    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451915    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451918    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451922    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451925    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451928    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451931    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451935    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451938    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451941    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451943    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:39:10.455501 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451946    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451948    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451951    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451953    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451956    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451959    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451961    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451963    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451966    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451969    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451972    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451974    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451977    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451979    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451982    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451985    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451988    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451990    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451993    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451995    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:39:10.455958 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.451997    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452000    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452071    2558 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452079    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452087    2558 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452093    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452100    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452104    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452110    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452116    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452121    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452125    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452129    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452133    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452136    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452139    2558 flags.go:64] FLAG: --cgroup-root=""
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452142    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452144    2558 flags.go:64] FLAG: --client-ca-file=""
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452148    2558 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452151    2558 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452155    2558 flags.go:64] FLAG: --cluster-domain=""
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452158    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452161    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452164    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 14:39:10.456444 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452167    2558 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452171    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452174    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452177    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452181    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452184    2558 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452187    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452190    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452193    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452196    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452200    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452203    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452206    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452208    2558 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452211    2558 flags.go:64] FLAG: --enable-server="true"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452214    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452219    2558 flags.go:64] FLAG: --event-burst="100"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452222    2558 flags.go:64] FLAG: --event-qps="50"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452225    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452228    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452230    2558 flags.go:64] FLAG: --eviction-hard=""
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452234    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452237    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452240    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452243    2558 flags.go:64] FLAG: --eviction-soft=""
Jul 01 14:39:10.457067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452246    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452249    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452255    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452258    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452260    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452263    2558 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452266    2558 flags.go:64] FLAG: --feature-gates=""
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452270    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452273    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452277    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452280    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452283    2558 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452286    2558 flags.go:64] FLAG: --help="false"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452289    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452292    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452294    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452297    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452301    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452304    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452307    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452310    2558 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452312    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452315    2558 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 14:39:10.457642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452318    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452321    2558 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452324    2558 flags.go:64] FLAG: --kube-reserved=""
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452327    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452329    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452332    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452335    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452337    2558 flags.go:64] FLAG: --lock-file=""
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452340    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452343    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452346    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452356    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452362    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452365    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452368    2558 flags.go:64] FLAG: --logging-format="text"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452371    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452375    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452377    2558 flags.go:64] FLAG: --manifest-url=""
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452380    2558 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452384    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452387    2558 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452391    2558 flags.go:64] FLAG: --max-pods="110"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452394    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452397    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452399    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 14:39:10.458258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452402    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452405    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452408    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452411    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452417    2558 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452420    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452423    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452426    2558 flags.go:64] FLAG: --pod-cidr=""
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452429    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452434    2558 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452436    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452439    2558 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452442    2558 flags.go:64] FLAG: --port="10250"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452445    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452450    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f59672b21d0e1e98"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452453    2558 flags.go:64] FLAG: --qos-reserved=""
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452457    2558 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452460    2558 flags.go:64] FLAG: --register-node="true"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452462    2558 flags.go:64] FLAG: --register-with-taints=""
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452469    2558 flags.go:64] FLAG: --registry-burst="10"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452477    2558 flags.go:64] FLAG: --registry-qps="5"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452480    2558 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452483    2558 flags.go:64] FLAG: --reserved-memory=""
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452487    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452490    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 14:39:10.458842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452493    2558 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452495    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452498    2558 flags.go:64] FLAG: --runonce="false"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452501    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452504    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452507    2558 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452509    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452512    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452515    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452518    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452521    2558 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452523    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452526    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452529    2558 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452532    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452535    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452538    2558 flags.go:64] FLAG: --system-cgroups=""
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452540    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452548    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452551    2558 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452554    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452558    2558 flags.go:64] FLAG: --tls-min-version=""
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452561    2558 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452564    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452567    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 14:39:10.459460 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452569    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452573    2558 flags.go:64] FLAG: --v="2"
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452577    2558 flags.go:64] FLAG: --version="false"
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452582    2558 flags.go:64] FLAG: --vmodule=""
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452586    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.452589    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452692    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452697    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452701    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452705    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452709    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452713    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452717    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452719    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452722    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452725    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452727    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452731    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452735    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452738    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452741    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:39:10.460107 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452744    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452747    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452750    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452752    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452755    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452757    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452760    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452765    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452768    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452771    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452773    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452776    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452779    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452782    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452784    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452788    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452790    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452793    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452795    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:39:10.460608 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452798    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452802    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452805    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452808    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452811    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452814    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452816    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452819    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452821    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452824    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452826    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452829    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452831    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452834    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452836    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452839    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452841    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452844    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452846    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:39:10.461094 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452848    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452852    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452855    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452857    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452859    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452862    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452865    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452867    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452869    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452872    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452875    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452878    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452880    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452883    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452885    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452888    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452890    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452893    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452895    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452910    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:39:10.461552 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452912    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452915    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452918    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452920    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452923    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452925    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452928    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452930    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452933    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452936    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452938    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452940    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452943    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452947    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452949    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452952    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452954    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452957    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452960    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:39:10.462137 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452963    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452966    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452968    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452970    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452973    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452976    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.452978    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.453535    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.459635    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.459649    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.459669    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.459676    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459727    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459731    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459735    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:39:10.462603 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459738    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459741    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459745    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459748    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459751    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459754    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459757    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459759    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459762    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459764    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459767    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459769    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459772    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459774    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459777    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459780    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459783    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459785    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459788    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:39:10.463003 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459790    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459793    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459795    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459798    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459801    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459803    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459806    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459810    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459813    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459816    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459820    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459824    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459827    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459830    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459832    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459835    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459838    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459840    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459843    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459845    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:39:10.463493 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459848    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459850    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459853    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459855    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459858    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459860    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459863    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459865    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459868    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459870    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459873    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459875    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459877    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459880    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459884    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459887    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459890    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459893    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459895    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:39:10.464004 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459912    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459915    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459918    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459921    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459924    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459927    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459930    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459932    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459935    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459937    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459940    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459943    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459945    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459948    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459951    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459953    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459956    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459958    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459961    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459964    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:39:10.464459 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459966    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459969    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459971    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459974    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459976    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459979    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459981    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459984    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459986    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459990    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459993    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.459996    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460000    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460002    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460006    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460008    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460011    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:39:10.464952 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460014    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.460019    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460138    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460143    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460146    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460149    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460152    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460155    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460157    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460160    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460163    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460165    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460168    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460170    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460173    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:39:10.465376 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460176    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460178    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460181    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460183    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460186    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460189    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460191    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460194    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460196    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460198    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460201    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460203    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460206    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460209    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460214    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460218    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460220    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460223    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460226    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460229    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:39:10.465746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460232    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460235    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460239    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460241    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460244    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460247    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460249    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460252    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460255    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460257    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460260    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460262    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460265    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460267    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460270    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460272    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460275    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460278    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460280    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:39:10.466312 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460282    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460285    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460288    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460290    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460293    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460295    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460299    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460303    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460306    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460309    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460312    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460315    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460318    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460321    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460323    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460326    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460329    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460331    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460334    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:39:10.466747 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460336    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460339    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460341    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460344    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460346    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460348    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460351    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460353    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460356    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460359    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460361    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460364    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460366    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460369    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460371    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460374    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460376    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460379    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460382    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460385    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:39:10.467219 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460387    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460390    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460392    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460395    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460398    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460400    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460403    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:10.460405    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.460410    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.460508    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.462360    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.463474    2558 server.go:1019] "Starting client certificate rotation"
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.463571    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 14:39:10.467687 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.463598    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 14:39:10.468017 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.465710    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 14:39:10.470768 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.470751    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 14:39:10.480456 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.480440    2558 log.go:25] "Validated CRI v1 runtime API"
Jul 01 14:39:10.482414 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.482399    2558 log.go:25] "Validated CRI v1 image API"
Jul 01 14:39:10.483263 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.483249    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 14:39:10.487319 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.487302    2558 fs.go:135] Filesystem UUIDs: map[6e0c5245-aa8b-4842-a286-b90967a412a3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 80f78a44-b4f6-4173-ac80-ee51bfac6c8d:/dev/nvme0n1p4]
Jul 01 14:39:10.487390 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.487319    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 14:39:10.494936 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.494799    2558 manager.go:216] Machine: {Timestamp:2026-07-01 14:39:10.49338507 +0000 UTC m=+0.311751017 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101312 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2636ebdaae03ac3958bd57478e1389 SystemUUID:ec2636eb-daae-03ac-3958-bd57478e1389 BootID:07e2052f-312b-44af-a21b-22c7a0f99963 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:80:5f:f2:7e:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:80:5f:f2:7e:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:77:ca:1b:80:6d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 14:39:10.494936 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.494927    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 14:39:10.495052 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.495002    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 14:39:10.497170 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.497154    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 14:39:10.498059 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.498032    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 14:39:10.498230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.498064    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-20.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 14:39:10.498931 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.498919    2558 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 14:39:10.498984 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.498937    2558 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 14:39:10.498984 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.498947    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 14:39:10.499645 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.499634    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 14:39:10.500658 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.500647    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 14:39:10.503719 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.503704    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 14:39:10.503859 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.503848    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 14:39:10.503930 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.503887    2558 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 14:39:10.503930 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.503919    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 14:39:10.504015 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.503940    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 14:39:10.504015 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.503953    2558 kubelet.go:398] "Adding apiserver pod source"
Jul 01 14:39:10.504015 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.503966    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 14:39:10.504330 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.504317    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 14:39:10.504390 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.504341    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 14:39:10.507233 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.507216    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 14:39:10.508682 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.508669    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 14:39:10.508735 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.508698    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 14:39:10.509288 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509275    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 14:39:10.509345 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509296    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 14:39:10.509345 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509306    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 14:39:10.509345 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509315    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 14:39:10.509345 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509326    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 14:39:10.509345 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509334    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 14:39:10.509485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509402    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 14:39:10.509485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509409    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 14:39:10.509485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509416    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 14:39:10.509485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509421    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 14:39:10.509485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509429    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 14:39:10.509485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.509436    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 14:39:10.510750 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.510739    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 14:39:10.510750 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.510750    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 14:39:10.514019 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.514006    2558 server.go:1268] "Started kubelet"
Jul 01 14:39:10.514161 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.514105    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 14:39:10.514161 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.514125    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 14:39:10.514263 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.514191    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 14:39:10.514829 ip-10-0-134-20 systemd[1]: Started Kubernetes Kubelet.
Jul 01 14:39:10.515485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.515466    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 14:39:10.515571 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.515532    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-20.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 14:39:10.515571 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.515531    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 14:39:10.515571 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.515553    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-20.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 14:39:10.517367 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.517353    2558 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 14:39:10.519344 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.518458    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-20.ec2.internal.18be31569a7acca5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-20.ec2.internal,UID:ip-10-0-134-20.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-20.ec2.internal,},FirstTimestamp:2026-07-01 14:39:10.513982629 +0000 UTC m=+0.332348577,LastTimestamp:2026-07-01 14:39:10.513982629 +0000 UTC m=+0.332348577,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-20.ec2.internal,}"
Jul 01 14:39:10.519807 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.519787    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 14:39:10.522051 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.522027    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 14:39:10.522051 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.522037    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 14:39:10.522199 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.522114    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:10.522860 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.522842    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 14:39:10.522860 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.522860    2558 factory.go:55] Registering systemd factory
Jul 01 14:39:10.523015 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.522868    2558 factory.go:223] Registration of the systemd container factory successfully
Jul 01 14:39:10.523015 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.522926    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 14:39:10.523015 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.522968    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 14:39:10.523015 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.522980    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 14:39:10.523168 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.523094    2558 factory.go:153] Registering CRI-O factory
Jul 01 14:39:10.523168 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.523108    2558 factory.go:223] Registration of the crio container factory successfully
Jul 01 14:39:10.523168 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.523106    2558 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 14:39:10.523168 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.523138    2558 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 14:39:10.523318 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.523130    2558 factory.go:103] Registering Raw factory
Jul 01 14:39:10.523318 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.523206    2558 manager.go:1196] Started watching for new ooms in manager
Jul 01 14:39:10.524394 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.524374    2558 manager.go:349] Starting recovery of all containers
Jul 01 14:39:10.526967 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.526943    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mlr4t"
Jul 01 14:39:10.534438 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.533854    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-20.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 14:39:10.534438 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.533926    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 14:39:10.537298 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.537281    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mlr4t"
Jul 01 14:39:10.537942 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.537918    2558 manager.go:354] Recovery completed
Jul 01 14:39:10.543054 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.543039    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:39:10.544769 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.544754    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:39:10.544840 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.544782    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:39:10.544840 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.544796    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:39:10.545644 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.545501    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 14:39:10.545644 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.545513    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 14:39:10.545644 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.545529    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 14:39:10.545644 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.545535    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:39:10.546367 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.546352    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:39:10.546438 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.546377    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:39:10.546438 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.546387    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:39:10.546535 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.546522    2558 policy_none.go:49] "None policy: Start"
Jul 01 14:39:10.546581 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.546541    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 14:39:10.546581 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.546552    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 14:39:10.547573 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.547493    2558 policy_none.go:47] "Start"
Jul 01 14:39:10.547999 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.547884    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-20.ec2.internal.18be31569c50931e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-20.ec2.internal,UID:ip-10-0-134-20.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-20.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-20.ec2.internal,},FirstTimestamp:2026-07-01 14:39:10.544769822 +0000 UTC m=+0.363135794,LastTimestamp:2026-07-01 14:39:10.544769822 +0000 UTC m=+0.363135794,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-20.ec2.internal,}"
Jul 01 14:39:10.588389 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.588363    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.589600    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.589641    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.589663    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.589701    2558 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.590721    2558 manager.go:337] "Starting Device Plugin manager"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.590757    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.590770    2558 server.go:87] "Starting device plugin registration server"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.591042    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.591053    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.591264    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.591396    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.591406    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.592866    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.592920    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:39:10.594245 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.592929    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:10.690087 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.690043    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal"]
Jul 01 14:39:10.690249 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.690146    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:39:10.691391 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.691377    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:39:10.691784 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.691766    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:39:10.691853 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.691801    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:39:10.691853 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.691817    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:39:10.692134 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.692116    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:39:10.692212 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.692145    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:39:10.692212 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.692156    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:39:10.692212 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.692178    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.693052 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.693040    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:39:10.693165 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.693151    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.693219 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.693192    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:39:10.693759 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.693742    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:39:10.693850 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.693775    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:39:10.693850 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.693793    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:39:10.693850 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.693802    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:39:10.693850 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.693823    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:39:10.693850 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.693835    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:39:10.695056 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.695009    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.695056 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.695034    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:39:10.695619 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.695607    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:39:10.695680 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.695628    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:39:10.695680 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.695637    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:39:10.701535 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.701519    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.701535 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.701537    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-20.ec2.internal\": node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:10.724186 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.724035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5389ab3ca8a898983720c30b1081103-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"a5389ab3ca8a898983720c30b1081103\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.725551 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.725536    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-20.ec2.internal\" not found" node="ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.728735 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.727883    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:10.730700 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.730683    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-20.ec2.internal\" not found" node="ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.824416 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.824391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5389ab3ca8a898983720c30b1081103-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"a5389ab3ca8a898983720c30b1081103\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.824553 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.824421    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bdd498e17258fb69bcdf5cab2dc7417-config\") pod \"kube-apiserver-proxy-ip-10-0-134-20.ec2.internal\" (UID: \"1bdd498e17258fb69bcdf5cab2dc7417\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.824553 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.824456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5389ab3ca8a898983720c30b1081103-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"a5389ab3ca8a898983720c30b1081103\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.824553 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.824501    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5389ab3ca8a898983720c30b1081103-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"a5389ab3ca8a898983720c30b1081103\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.828507 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.828491    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:10.924982 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.924952    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5389ab3ca8a898983720c30b1081103-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"a5389ab3ca8a898983720c30b1081103\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.924982 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.924983    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bdd498e17258fb69bcdf5cab2dc7417-config\") pod \"kube-apiserver-proxy-ip-10-0-134-20.ec2.internal\" (UID: \"1bdd498e17258fb69bcdf5cab2dc7417\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.925151 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.925025    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bdd498e17258fb69bcdf5cab2dc7417-config\") pod \"kube-apiserver-proxy-ip-10-0-134-20.ec2.internal\" (UID: \"1bdd498e17258fb69bcdf5cab2dc7417\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.925151 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:10.925050    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5389ab3ca8a898983720c30b1081103-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal\" (UID: \"a5389ab3ca8a898983720c30b1081103\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:10.929075 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:10.929061    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:11.028939 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:11.028865    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:11.029940 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:11.029924    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:11.033929 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:11.033915    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:11.130464 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:11.130436    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:11.231008 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:11.230985    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:13.089408 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.089351    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:13.190068 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.190041    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-20.ec2.internal\" not found"
Jul 01 14:39:13.221223 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.221203    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 14:39:13.221578 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.221344    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 14:39:13.245180 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.245163    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:39:13.262188 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.262165    2558 apiserver.go:52] "Watching apiserver"
Jul 01 14:39:13.271903 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.271882    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 14:39:13.273296 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.273276    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc","openshift-cluster-node-tuning-operator/tuned-pms96","openshift-dns/node-resolver-4lrpd","openshift-image-registry/node-ca-ztcz2","openshift-multus/multus-additional-cni-plugins-bwvqg","openshift-ovn-kubernetes/ovnkube-node-jtdlp","kube-system/konnectivity-agent-phzkb","openshift-multus/multus-5cqvx","openshift-multus/network-metrics-daemon-khdj5","openshift-network-diagnostics/network-check-target-llw6c","openshift-network-operator/iptables-alerter-txzp9","kube-system/global-pull-secret-syncer-t85xw"]
Jul 01 14:39:13.274799 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.274779    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.275931 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.275913    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.276874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.276852    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 14:39:13.276963 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.276908    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 14:39:13.276963 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.276944    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 14:39:13.276963 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.276952    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cdt7b\""
Jul 01 14:39:13.277107 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.277013    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.277842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.277813    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 14:39:13.278027 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.278014    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.278161 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.278144    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 14:39:13.278414 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.278400    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ch2cb\""
Jul 01 14:39:13.279111 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.279097    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 14:39:13.279221 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.279208    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.279272 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.279248    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gfdpf\""
Jul 01 14:39:13.279333 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.279318    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 14:39:13.280236 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.279899    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 14:39:13.280236 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.279979    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:13.280465 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.280440    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 14:39:13.280597 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.280522    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7g7tm\""
Jul 01 14:39:13.280597 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.280578    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 14:39:13.280749 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.280634    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 14:39:13.280978 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.280963    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.281885 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.281866    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 14:39:13.281885 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.281885    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 14:39:13.282089 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.282076    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 14:39:13.282148 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.282100    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:13.282357 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.282333    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 14:39:13.282776 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.282755    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cn4c2\""
Jul 01 14:39:13.283187 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.283168    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 14:39:13.283551 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.283536    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.285006    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.285065    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.285073    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.285487    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.285939    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.285982    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ztnkr\""
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286104    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286184    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286273    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-log-socket\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286356    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-host\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286383    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n78d\" (UniqueName: \"kubernetes.io/projected/7abab760-11aa-4e86-90df-31cbfb591873-kube-api-access-5n78d\") pod \"node-resolver-4lrpd\" (UID: \"7abab760-11aa-4e86-90df-31cbfb591873\") " pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.286483 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286413    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-system-cni-dir\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-cnibin\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286567    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-slash\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-run-ovn\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286624    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286657    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-lib-modules\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286704    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286766    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286819    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286823    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-sys-fs\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286866    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-cni-bin\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpl24\" (UniqueName: \"kubernetes.io/projected/95dbc66a-5975-4106-ba5d-9564565d86f2-kube-api-access-qpl24\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-device-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.286951    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.286980    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-sysctl-conf\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-var-lib-kubelet\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287030    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 14:39:13.287076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7abab760-11aa-4e86-90df-31cbfb591873-hosts-file\") pod \"node-resolver-4lrpd\" (UID: \"7abab760-11aa-4e86-90df-31cbfb591873\") " pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287058    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b05de0b-5d43-4566-b311-699a60bbd0cf-host\") pod \"node-ca-ztcz2\" (UID: \"0b05de0b-5d43-4566-b311-699a60bbd0cf\") " pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287087    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93e55270-6d6b-40c6-b27b-20c14c438f0e-tmp\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abab760-11aa-4e86-90df-31cbfb591873-tmp-dir\") pod \"node-resolver-4lrpd\" (UID: \"7abab760-11aa-4e86-90df-31cbfb591873\") " pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287144    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95dbc66a-5975-4106-ba5d-9564565d86f2-ovn-node-metrics-cert\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49pxd\" (UniqueName: \"kubernetes.io/projected/b688282d-3143-4fc3-94ab-30cdc3bc57d9-kube-api-access-49pxd\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-kubelet\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287301    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-run-netns\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287367    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-modprobe-d\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b05de0b-5d43-4566-b311-699a60bbd0cf-serviceca\") pod \"node-ca-ztcz2\" (UID: \"0b05de0b-5d43-4566-b311-699a60bbd0cf\") " pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.287883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287850    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rwf4b\""
Jul 01 14:39:13.288386 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.287917    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 14:39:13.288386 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.288029    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95dbc66a-5975-4106-ba5d-9564565d86f2-ovnkube-config\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.288504 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.288483    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9abca8ef-87fb-4559-a1ec-f4fd48071abf-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.288709 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.288567    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-cni-netd\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.288709 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.288601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95dbc66a-5975-4106-ba5d-9564565d86f2-ovnkube-script-lib\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.288709 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.288674    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5sg5n\""
Jul 01 14:39:13.288859 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.288719    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-registration-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.288859 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.288754    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-tuned\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.289004 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.288947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f688x\" (UniqueName: \"kubernetes.io/projected/9abca8ef-87fb-4559-a1ec-f4fd48071abf-kube-api-access-f688x\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.289472 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.289154    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-systemd-units\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.289472 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.289382    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-systemd\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.289472 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.289419    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-os-release\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.289472 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.289454    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9abca8ef-87fb-4559-a1ec-f4fd48071abf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.289927 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.289494    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-etc-openvswitch\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.290142 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.289569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95dbc66a-5975-4106-ba5d-9564565d86f2-env-overrides\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.290142 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290043    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-kubernetes\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.290142 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290077    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-sysctl-d\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.290142 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290099    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhd29\" (UniqueName: \"kubernetes.io/projected/0b05de0b-5d43-4566-b311-699a60bbd0cf-kube-api-access-xhd29\") pod \"node-ca-ztcz2\" (UID: \"0b05de0b-5d43-4566-b311-699a60bbd0cf\") " pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.290142 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9abca8ef-87fb-4559-a1ec-f4fd48071abf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.290142 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290144    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-node-log\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.290428 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290166    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.290428 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290190    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-socket-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.290428 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290211    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-sysconfig\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.290428 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290234    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-run-openvswitch\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.290428 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290254    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-run\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.290428 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290273    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-sys\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.290428 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290294    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-var-lib-openvswitch\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.290428 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290314    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-457hr\" (UniqueName: \"kubernetes.io/projected/93e55270-6d6b-40c6-b27b-20c14c438f0e-kube-api-access-457hr\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.290428 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.290333    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-run-systemd\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.292344 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.292323    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:13.292440 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.292355    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 14:39:13.292440 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.292396    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:13.294199 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.294183    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.295576 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.295550    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:13.295676 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.295610    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:13.297080 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.296834    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 14:39:13.297080 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.296896    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wwlhj\""
Jul 01 14:39:13.297402 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.297380    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 14:39:13.297553 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.297541    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 14:39:13.298531 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.298488    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 14:34:12 +0000 UTC" deadline="2027-12-09 07:28:27.552838209 +0000 UTC"
Jul 01 14:39:13.298594 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.298531    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12616h49m14.254310168s"
Jul 01 14:39:13.298767 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.298752    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 14:39:13.298827 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.298811    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal"
Jul 01 14:39:13.298925 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.298910    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal"]
Jul 01 14:39:13.310739 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.310724    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 14:39:13.310812 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.310772    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal"]
Jul 01 14:39:13.316243 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.316226    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gn7r9"
Jul 01 14:39:13.323599 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.323579    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gn7r9"
Jul 01 14:39:13.341786 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:13.341729    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdd498e17258fb69bcdf5cab2dc7417.slice/crio-d80090d74782eed496be037f7a26b47da0dc11bb8c7124c3a13f4410d5351a85 WatchSource:0}: Error finding container d80090d74782eed496be037f7a26b47da0dc11bb8c7124c3a13f4410d5351a85: Status 404 returned error can't find the container with id d80090d74782eed496be037f7a26b47da0dc11bb8c7124c3a13f4410d5351a85
Jul 01 14:39:13.345754 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.345740    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 14:39:13.350408 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.350376    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" event={"ID":"1bdd498e17258fb69bcdf5cab2dc7417","Type":"ContainerStarted","Data":"d80090d74782eed496be037f7a26b47da0dc11bb8c7124c3a13f4410d5351a85"}
Jul 01 14:39:13.382355 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.382335    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 14:39:13.390967 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.390947    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-log-socket\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391034 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.390969    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-host\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.391034 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.390986    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n78d\" (UniqueName: \"kubernetes.io/projected/7abab760-11aa-4e86-90df-31cbfb591873-kube-api-access-5n78d\") pod \"node-resolver-4lrpd\" (UID: \"7abab760-11aa-4e86-90df-31cbfb591873\") " pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.391034 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391003    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-system-cni-dir\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.391034 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-cnibin\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-run-netns\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391051    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-host\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391069    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-etc-kubernetes\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391073    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-log-socket\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391085    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-system-cni-dir\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391111    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391108    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-cnibin\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-slash\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391155    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-run-ovn\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391165    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-slash\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391175    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-lib-modules\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391205    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-run-ovn\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391230 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391208    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391244    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391264    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391270    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391294    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-sys-fs\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391311    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-lib-modules\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8bnk\" (UniqueName: \"kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk\") pod \"network-check-target-llw6c\" (UID: \"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c\") " pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391342    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391362    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391384    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-cni-bin\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391422    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-cni-bin\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpl24\" (UniqueName: \"kubernetes.io/projected/95dbc66a-5975-4106-ba5d-9564565d86f2-kube-api-access-qpl24\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391452    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-device-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391492    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-sysctl-conf\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391515    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-var-lib-kubelet\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.391762 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391569    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-var-lib-kubelet\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391568    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-device-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391951    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7abab760-11aa-4e86-90df-31cbfb591873-hosts-file\") pod \"node-resolver-4lrpd\" (UID: \"7abab760-11aa-4e86-90df-31cbfb591873\") " pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392029    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-sysctl-conf\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392064    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b05de0b-5d43-4566-b311-699a60bbd0cf-host\") pod \"node-ca-ztcz2\" (UID: \"0b05de0b-5d43-4566-b311-699a60bbd0cf\") " pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.391361    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-sys-fs\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392102    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7abab760-11aa-4e86-90df-31cbfb591873-hosts-file\") pod \"node-resolver-4lrpd\" (UID: \"7abab760-11aa-4e86-90df-31cbfb591873\") " pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392155    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b05de0b-5d43-4566-b311-699a60bbd0cf-host\") pod \"node-ca-ztcz2\" (UID: \"0b05de0b-5d43-4566-b311-699a60bbd0cf\") " pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392166    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0e28ac5-ab10-4108-b51f-185ea52ba4f2-konnectivity-ca\") pod \"konnectivity-agent-phzkb\" (UID: \"e0e28ac5-ab10-4108-b51f-185ea52ba4f2\") " pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93e55270-6d6b-40c6-b27b-20c14c438f0e-tmp\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abab760-11aa-4e86-90df-31cbfb591873-tmp-dir\") pod \"node-resolver-4lrpd\" (UID: \"7abab760-11aa-4e86-90df-31cbfb591873\") " pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392315    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcptn\" (UniqueName: \"kubernetes.io/projected/9120aa3b-1ffb-484c-9471-601a7b69a1d5-kube-api-access-dcptn\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.392356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392359    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95dbc66a-5975-4106-ba5d-9564565d86f2-ovn-node-metrics-cert\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.392921 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392393    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49pxd\" (UniqueName: \"kubernetes.io/projected/b688282d-3143-4fc3-94ab-30cdc3bc57d9-kube-api-access-49pxd\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.392921 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392497    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 14:39:13.392921 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392680    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abab760-11aa-4e86-90df-31cbfb591873-tmp-dir\") pod \"node-resolver-4lrpd\" (UID: \"7abab760-11aa-4e86-90df-31cbfb591873\") " pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.392921 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392842    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.392921 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392906    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-cnibin\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.393146 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.392940    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-os-release\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.393146 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393004    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-var-lib-cni-bin\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.393146 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393043    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0e28ac5-ab10-4108-b51f-185ea52ba4f2-agent-certs\") pod \"konnectivity-agent-phzkb\" (UID: \"e0e28ac5-ab10-4108-b51f-185ea52ba4f2\") " pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:13.393146 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-cni-dir\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.393146 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393126    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.393357 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393137    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-run-k8s-cni-cncf-io\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.393357 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393241    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-hostroot\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.393357 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-kubelet\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.393487 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393356    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-run-netns\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.393487 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393366    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-kubelet\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.393487 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-modprobe-d\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.393611 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b05de0b-5d43-4566-b311-699a60bbd0cf-serviceca\") pod \"node-ca-ztcz2\" (UID: \"0b05de0b-5d43-4566-b311-699a60bbd0cf\") " pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.393611 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393584    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-run-multus-certs\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.393705 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95dbc66a-5975-4106-ba5d-9564565d86f2-ovnkube-config\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.393759 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393720    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9abca8ef-87fb-4559-a1ec-f4fd48071abf-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.393759 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cd6a69f-c352-40c4-a8b9-125145b3ec48-kubelet-config\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:13.393862 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393780    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-var-lib-cni-multus\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.393862 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-cni-netd\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.393862 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95dbc66a-5975-4106-ba5d-9564565d86f2-ovnkube-script-lib\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.394003 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393906    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-registration-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.394003 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393935    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-tuned\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.394003 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.393983    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f688x\" (UniqueName: \"kubernetes.io/projected/9abca8ef-87fb-4559-a1ec-f4fd48071abf-kube-api-access-f688x\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.394134 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394012    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-systemd-units\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.394134 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394039    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-systemd\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.394134 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-os-release\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.394134 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394094    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9abca8ef-87fb-4559-a1ec-f4fd48071abf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.394134 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-system-cni-dir\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.394370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394154    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x4fm\" (UniqueName: \"kubernetes.io/projected/388279eb-7ee2-4130-9f63-3f54e10c7419-kube-api-access-9x4fm\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:13.394370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-etc-openvswitch\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.394370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394209    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95dbc66a-5975-4106-ba5d-9564565d86f2-env-overrides\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.394559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394389    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-kubernetes\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.394559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394416    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-sysctl-d\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.394559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394453    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhd29\" (UniqueName: \"kubernetes.io/projected/0b05de0b-5d43-4566-b311-699a60bbd0cf-kube-api-access-xhd29\") pod \"node-ca-ztcz2\" (UID: \"0b05de0b-5d43-4566-b311-699a60bbd0cf\") " pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.394559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9abca8ef-87fb-4559-a1ec-f4fd48071abf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.394559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394517    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-conf-dir\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.394559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394546    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-node-log\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.394874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394577    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.394874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-socket-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.394874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394628    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-sysconfig\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.394874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394657    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cd6a69f-c352-40c4-a8b9-125145b3ec48-dbus\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:13.394874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394703    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-socket-dir-parent\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.394874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-var-lib-kubelet\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.394874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394765    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-daemon-config\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.394874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-run-openvswitch\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.394874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394825    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-run\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.394874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394851    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-sys\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.395317 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394881    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqx8\" (UniqueName: \"kubernetes.io/projected/35f35078-e173-4181-9dbc-ade18144ebd6-kube-api-access-lsqx8\") pod \"iptables-alerter-txzp9\" (UID: \"35f35078-e173-4181-9dbc-ade18144ebd6\") " pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.395317 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394906    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-var-lib-openvswitch\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.395317 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394936    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-457hr\" (UniqueName: \"kubernetes.io/projected/93e55270-6d6b-40c6-b27b-20c14c438f0e-kube-api-access-457hr\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.395317 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394965    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/35f35078-e173-4181-9dbc-ade18144ebd6-iptables-alerter-script\") pod \"iptables-alerter-txzp9\" (UID: \"35f35078-e173-4181-9dbc-ade18144ebd6\") " pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.395317 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.394996    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9120aa3b-1ffb-484c-9471-601a7b69a1d5-cni-binary-copy\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.395317 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.395190    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-run-systemd\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.395317 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.395221    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35f35078-e173-4181-9dbc-ade18144ebd6-host-slash\") pod \"iptables-alerter-txzp9\" (UID: \"35f35078-e173-4181-9dbc-ade18144ebd6\") " pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.395763 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.395726    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/95dbc66a-5975-4106-ba5d-9564565d86f2-env-overrides\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.395866 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.395851    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-sysctl-d\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.395984 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.395968    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-kubernetes\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.396054 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.396028    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-cni-netd\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.396279 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.396144    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-systemd\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.396379 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.396303    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-etc-openvswitch\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.396465 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.396444    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b05de0b-5d43-4566-b311-699a60bbd0cf-serviceca\") pod \"node-ca-ztcz2\" (UID: \"0b05de0b-5d43-4566-b311-699a60bbd0cf\") " pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.396861 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.396842    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9abca8ef-87fb-4559-a1ec-f4fd48071abf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.396861 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.396850    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9abca8ef-87fb-4559-a1ec-f4fd48071abf-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.397031 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.396220    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-registration-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.397031 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.396922    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-modprobe-d\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.397183 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397056    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93e55270-6d6b-40c6-b27b-20c14c438f0e-tmp\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.397183 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397107    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-systemd-units\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.397183 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397130    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/95dbc66a-5975-4106-ba5d-9564565d86f2-ovn-node-metrics-cert\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397193    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/95dbc66a-5975-4106-ba5d-9564565d86f2-ovnkube-script-lib\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.396254    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9abca8ef-87fb-4559-a1ec-f4fd48071abf-os-release\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397227    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-sysconfig\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397245    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b688282d-3143-4fc3-94ab-30cdc3bc57d9-socket-dir\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397259    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-node-log\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397278    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397311    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-var-lib-openvswitch\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397331    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-sys\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397352    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-run-openvswitch\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397364    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/95dbc66a-5975-4106-ba5d-9564565d86f2-ovnkube-config\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.397401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397400    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93e55270-6d6b-40c6-b27b-20c14c438f0e-run\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.397929 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397443    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-host-run-netns\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.397929 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397539    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/95dbc66a-5975-4106-ba5d-9564565d86f2-run-systemd\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.397929 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.397550    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9abca8ef-87fb-4559-a1ec-f4fd48071abf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.398248 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.398230    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93e55270-6d6b-40c6-b27b-20c14c438f0e-etc-tuned\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.400554 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.400532    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n78d\" (UniqueName: \"kubernetes.io/projected/7abab760-11aa-4e86-90df-31cbfb591873-kube-api-access-5n78d\") pod \"node-resolver-4lrpd\" (UID: \"7abab760-11aa-4e86-90df-31cbfb591873\") " pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.400883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.400868    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49pxd\" (UniqueName: \"kubernetes.io/projected/b688282d-3143-4fc3-94ab-30cdc3bc57d9-kube-api-access-49pxd\") pod \"aws-ebs-csi-driver-node-6lzxc\" (UID: \"b688282d-3143-4fc3-94ab-30cdc3bc57d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.401839 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.401820    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpl24\" (UniqueName: \"kubernetes.io/projected/95dbc66a-5975-4106-ba5d-9564565d86f2-kube-api-access-qpl24\") pod \"ovnkube-node-jtdlp\" (UID: \"95dbc66a-5975-4106-ba5d-9564565d86f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.407489 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.407466    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f688x\" (UniqueName: \"kubernetes.io/projected/9abca8ef-87fb-4559-a1ec-f4fd48071abf-kube-api-access-f688x\") pod \"multus-additional-cni-plugins-bwvqg\" (UID: \"9abca8ef-87fb-4559-a1ec-f4fd48071abf\") " pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.408336 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.408320    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhd29\" (UniqueName: \"kubernetes.io/projected/0b05de0b-5d43-4566-b311-699a60bbd0cf-kube-api-access-xhd29\") pod \"node-ca-ztcz2\" (UID: \"0b05de0b-5d43-4566-b311-699a60bbd0cf\") " pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.408426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.408411    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-457hr\" (UniqueName: \"kubernetes.io/projected/93e55270-6d6b-40c6-b27b-20c14c438f0e-kube-api-access-457hr\") pod \"tuned-pms96\" (UID: \"93e55270-6d6b-40c6-b27b-20c14c438f0e\") " pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.496195 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bnk\" (UniqueName: \"kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk\") pod \"network-check-target-llw6c\" (UID: \"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c\") " pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:13.496195 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496193    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:13.496366 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496211    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0e28ac5-ab10-4108-b51f-185ea52ba4f2-konnectivity-ca\") pod \"konnectivity-agent-phzkb\" (UID: \"e0e28ac5-ab10-4108-b51f-185ea52ba4f2\") " pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:13.496366 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcptn\" (UniqueName: \"kubernetes.io/projected/9120aa3b-1ffb-484c-9471-601a7b69a1d5-kube-api-access-dcptn\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496366 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496281    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-cnibin\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496366 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.496302    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:13.496366 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-os-release\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.496379    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret podName:3cd6a69f-c352-40c4-a8b9-125145b3ec48 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:13.996341595 +0000 UTC m=+2.056920763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret") pod "global-pull-secret-syncer-t85xw" (UID: "3cd6a69f-c352-40c4-a8b9-125145b3ec48") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496381    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-os-release\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496414    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-var-lib-cni-bin\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496423    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-cnibin\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0e28ac5-ab10-4108-b51f-185ea52ba4f2-agent-certs\") pod \"konnectivity-agent-phzkb\" (UID: \"e0e28ac5-ab10-4108-b51f-185ea52ba4f2\") " pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496455    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-var-lib-cni-bin\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-cni-dir\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-run-k8s-cni-cncf-io\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496494    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-hostroot\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496512    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-run-multus-certs\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496530    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cd6a69f-c352-40c4-a8b9-125145b3ec48-kubelet-config\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496544    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-var-lib-cni-multus\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496565    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-system-cni-dir\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496575    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-hostroot\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.496589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496584    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-cni-dir\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496591    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x4fm\" (UniqueName: \"kubernetes.io/projected/388279eb-7ee2-4130-9f63-3f54e10c7419-kube-api-access-9x4fm\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496618    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-run-k8s-cni-cncf-io\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496593    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-run-multus-certs\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496628    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-var-lib-cni-multus\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-conf-dir\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496670    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-conf-dir\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496671    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cd6a69f-c352-40c4-a8b9-125145b3ec48-kubelet-config\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cd6a69f-c352-40c4-a8b9-125145b3ec48-dbus\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496728    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cd6a69f-c352-40c4-a8b9-125145b3ec48-dbus\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-socket-dir-parent\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496732    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-system-cni-dir\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496762    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-var-lib-kubelet\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496786    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-socket-dir-parent\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496790    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-daemon-config\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496803    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0e28ac5-ab10-4108-b51f-185ea52ba4f2-konnectivity-ca\") pod \"konnectivity-agent-phzkb\" (UID: \"e0e28ac5-ab10-4108-b51f-185ea52ba4f2\") " pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496819    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqx8\" (UniqueName: \"kubernetes.io/projected/35f35078-e173-4181-9dbc-ade18144ebd6-kube-api-access-lsqx8\") pod \"iptables-alerter-txzp9\" (UID: \"35f35078-e173-4181-9dbc-ade18144ebd6\") " pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496840    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-var-lib-kubelet\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/35f35078-e173-4181-9dbc-ade18144ebd6-iptables-alerter-script\") pod \"iptables-alerter-txzp9\" (UID: \"35f35078-e173-4181-9dbc-ade18144ebd6\") " pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496885    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9120aa3b-1ffb-484c-9471-601a7b69a1d5-cni-binary-copy\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35f35078-e173-4181-9dbc-ade18144ebd6-host-slash\") pod \"iptables-alerter-txzp9\" (UID: \"35f35078-e173-4181-9dbc-ade18144ebd6\") " pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-run-netns\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.496960    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-etc-kubernetes\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.497003    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-etc-kubernetes\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.497018    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9120aa3b-1ffb-484c-9471-601a7b69a1d5-host-run-netns\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.497004    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35f35078-e173-4181-9dbc-ade18144ebd6-host-slash\") pod \"iptables-alerter-txzp9\" (UID: \"35f35078-e173-4181-9dbc-ade18144ebd6\") " pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.497058    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.497162    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.497212    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs podName:388279eb-7ee2-4130-9f63-3f54e10c7419 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:13.997197066 +0000 UTC m=+2.057776236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs") pod "network-metrics-daemon-khdj5" (UID: "388279eb-7ee2-4130-9f63-3f54e10c7419") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.497326    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9120aa3b-1ffb-484c-9471-601a7b69a1d5-multus-daemon-config\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.497327    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/35f35078-e173-4181-9dbc-ade18144ebd6-iptables-alerter-script\") pod \"iptables-alerter-txzp9\" (UID: \"35f35078-e173-4181-9dbc-ade18144ebd6\") " pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.497923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.497472    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9120aa3b-1ffb-484c-9471-601a7b69a1d5-cni-binary-copy\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.498674 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.498655    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0e28ac5-ab10-4108-b51f-185ea52ba4f2-agent-certs\") pod \"konnectivity-agent-phzkb\" (UID: \"e0e28ac5-ab10-4108-b51f-185ea52ba4f2\") " pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:13.500258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.500242    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:39:13.504706 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.504678    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:39:13.504706 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.504708    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:39:13.504855 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.504717    2558 projected.go:196] Error preparing data for projected volume kube-api-access-l8bnk for pod openshift-network-diagnostics/network-check-target-llw6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:13.504855 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:13.504762    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk podName:c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c nodeName:}" failed. No retries permitted until 2026-07-01 14:39:14.00474937 +0000 UTC m=+2.065328542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l8bnk" (UniqueName: "kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk") pod "network-check-target-llw6c" (UID: "c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:13.505301 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.505283    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqx8\" (UniqueName: \"kubernetes.io/projected/35f35078-e173-4181-9dbc-ade18144ebd6-kube-api-access-lsqx8\") pod \"iptables-alerter-txzp9\" (UID: \"35f35078-e173-4181-9dbc-ade18144ebd6\") " pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.505513 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.505497    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcptn\" (UniqueName: \"kubernetes.io/projected/9120aa3b-1ffb-484c-9471-601a7b69a1d5-kube-api-access-dcptn\") pod \"multus-5cqvx\" (UID: \"9120aa3b-1ffb-484c-9471-601a7b69a1d5\") " pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.505754 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.505740    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x4fm\" (UniqueName: \"kubernetes.io/projected/388279eb-7ee2-4130-9f63-3f54e10c7419-kube-api-access-9x4fm\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:13.592696 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.592623    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc"
Jul 01 14:39:13.600793 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:13.600770    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb688282d_3143_4fc3_94ab_30cdc3bc57d9.slice/crio-179a8cc7a50213fb2279de30eaf76491b2e211c4a015eed7b0d7342415edacd5 WatchSource:0}: Error finding container 179a8cc7a50213fb2279de30eaf76491b2e211c4a015eed7b0d7342415edacd5: Status 404 returned error can't find the container with id 179a8cc7a50213fb2279de30eaf76491b2e211c4a015eed7b0d7342415edacd5
Jul 01 14:39:13.614298 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.614275    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pms96"
Jul 01 14:39:13.621627 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.621611    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4lrpd"
Jul 01 14:39:13.621873 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:13.621856    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e55270_6d6b_40c6_b27b_20c14c438f0e.slice/crio-77bdb4bd9d0049198052b5aee4f962180181253a0a791e628babc0da419d7efa WatchSource:0}: Error finding container 77bdb4bd9d0049198052b5aee4f962180181253a0a791e628babc0da419d7efa: Status 404 returned error can't find the container with id 77bdb4bd9d0049198052b5aee4f962180181253a0a791e628babc0da419d7efa
Jul 01 14:39:13.628726 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:13.628707    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abab760_11aa_4e86_90df_31cbfb591873.slice/crio-b82d2abe45a0b67b03b1940efc59e04894eb82ed0ec43bce0cfb0b1eea11b71e WatchSource:0}: Error finding container b82d2abe45a0b67b03b1940efc59e04894eb82ed0ec43bce0cfb0b1eea11b71e: Status 404 returned error can't find the container with id b82d2abe45a0b67b03b1940efc59e04894eb82ed0ec43bce0cfb0b1eea11b71e
Jul 01 14:39:13.638617 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.638602    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztcz2"
Jul 01 14:39:13.644778 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:13.644752    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b05de0b_5d43_4566_b311_699a60bbd0cf.slice/crio-f46e1dc65ab9439021921a53ee72146465be927e607103c47de0be0bc100f67b WatchSource:0}: Error finding container f46e1dc65ab9439021921a53ee72146465be927e607103c47de0be0bc100f67b: Status 404 returned error can't find the container with id f46e1dc65ab9439021921a53ee72146465be927e607103c47de0be0bc100f67b
Jul 01 14:39:13.647536 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.647522    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bwvqg"
Jul 01 14:39:13.650221 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.650203    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:39:13.653456 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.653440    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:13.654063 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:13.654048    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abca8ef_87fb_4559_a1ec_f4fd48071abf.slice/crio-75e94c0fdcac9f96f47a2193dc400a64d00bb414ddbda7eb93230a7f090b0750 WatchSource:0}: Error finding container 75e94c0fdcac9f96f47a2193dc400a64d00bb414ddbda7eb93230a7f090b0750: Status 404 returned error can't find the container with id 75e94c0fdcac9f96f47a2193dc400a64d00bb414ddbda7eb93230a7f090b0750
Jul 01 14:39:13.659287 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.659269    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:13.663961 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:13.663942    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95dbc66a_5975_4106_ba5d_9564565d86f2.slice/crio-ccb79a2fd34ba80ecb0652384380e4b7b90d7356d5a0b317f9fe05511011209e WatchSource:0}: Error finding container ccb79a2fd34ba80ecb0652384380e4b7b90d7356d5a0b317f9fe05511011209e: Status 404 returned error can't find the container with id ccb79a2fd34ba80ecb0652384380e4b7b90d7356d5a0b317f9fe05511011209e
Jul 01 14:39:13.665133 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.665110    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5cqvx"
Jul 01 14:39:13.666652 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:13.666632    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e28ac5_ab10_4108_b51f_185ea52ba4f2.slice/crio-fe8ec3992e9011da12c1490068de33ddc448b1787cafb46840a1a22830f542ae WatchSource:0}: Error finding container fe8ec3992e9011da12c1490068de33ddc448b1787cafb46840a1a22830f542ae: Status 404 returned error can't find the container with id fe8ec3992e9011da12c1490068de33ddc448b1787cafb46840a1a22830f542ae
Jul 01 14:39:13.669529 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:13.669515    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-txzp9"
Jul 01 14:39:13.672746 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:13.672726    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9120aa3b_1ffb_484c_9471_601a7b69a1d5.slice/crio-6b87d1622178c65a45adc397ef8bb1187bf571246a36d046ca654f8d5b6682bf WatchSource:0}: Error finding container 6b87d1622178c65a45adc397ef8bb1187bf571246a36d046ca654f8d5b6682bf: Status 404 returned error can't find the container with id 6b87d1622178c65a45adc397ef8bb1187bf571246a36d046ca654f8d5b6682bf
Jul 01 14:39:13.676946 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:13.676924    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f35078_e173_4181_9dbc_ade18144ebd6.slice/crio-7ea2d3dd3434fee991fab14772ef5244cb6a67d828c95fa2ca1a99c543f1d9ec WatchSource:0}: Error finding container 7ea2d3dd3434fee991fab14772ef5244cb6a67d828c95fa2ca1a99c543f1d9ec: Status 404 returned error can't find the container with id 7ea2d3dd3434fee991fab14772ef5244cb6a67d828c95fa2ca1a99c543f1d9ec
Jul 01 14:39:14.000785 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.000682    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:14.000929 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.000804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:14.001054 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:14.000932    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:14.001054 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:14.000944    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:14.001054 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:14.001001    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs podName:388279eb-7ee2-4130-9f63-3f54e10c7419 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:15.000982733 +0000 UTC m=+3.061561940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs") pod "network-metrics-daemon-khdj5" (UID: "388279eb-7ee2-4130-9f63-3f54e10c7419") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:14.001054 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:14.001022    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret podName:3cd6a69f-c352-40c4-a8b9-125145b3ec48 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:15.001010622 +0000 UTC m=+3.061589802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret") pod "global-pull-secret-syncer-t85xw" (UID: "3cd6a69f-c352-40c4-a8b9-125145b3ec48") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:14.101623 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.101590    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bnk\" (UniqueName: \"kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk\") pod \"network-check-target-llw6c\" (UID: \"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c\") " pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:14.101886 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:14.101794    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:39:14.101886 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:14.101815    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:39:14.101886 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:14.101828    2558 projected.go:196] Error preparing data for projected volume kube-api-access-l8bnk for pod openshift-network-diagnostics/network-check-target-llw6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:14.101886 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:14.101883    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk podName:c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c nodeName:}" failed. No retries permitted until 2026-07-01 14:39:15.10186606 +0000 UTC m=+3.162445232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8bnk" (UniqueName: "kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk") pod "network-check-target-llw6c" (UID: "c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:14.193071 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:39:14.193031    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5389ab3ca8a898983720c30b1081103.slice/crio-c94f7299865cf0acf969482f3f6a96ab96119beea3182d12dd502709305d201a WatchSource:0}: Error finding container c94f7299865cf0acf969482f3f6a96ab96119beea3182d12dd502709305d201a: Status 404 returned error can't find the container with id c94f7299865cf0acf969482f3f6a96ab96119beea3182d12dd502709305d201a
Jul 01 14:39:14.325498 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.325337    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 14:34:13 +0000 UTC" deadline="2028-02-14 19:40:35.645495454 +0000 UTC"
Jul 01 14:39:14.325498 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.325400    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14237h1m21.320099272s"
Jul 01 14:39:14.360150 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.360111    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" event={"ID":"a5389ab3ca8a898983720c30b1081103","Type":"ContainerStarted","Data":"c94f7299865cf0acf969482f3f6a96ab96119beea3182d12dd502709305d201a"}
Jul 01 14:39:14.363134 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.363057    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-txzp9" event={"ID":"35f35078-e173-4181-9dbc-ade18144ebd6","Type":"ContainerStarted","Data":"7ea2d3dd3434fee991fab14772ef5244cb6a67d828c95fa2ca1a99c543f1d9ec"}
Jul 01 14:39:14.365853 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.365743    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" event={"ID":"95dbc66a-5975-4106-ba5d-9564565d86f2","Type":"ContainerStarted","Data":"ccb79a2fd34ba80ecb0652384380e4b7b90d7356d5a0b317f9fe05511011209e"}
Jul 01 14:39:14.367992 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.367967    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztcz2" event={"ID":"0b05de0b-5d43-4566-b311-699a60bbd0cf","Type":"ContainerStarted","Data":"f46e1dc65ab9439021921a53ee72146465be927e607103c47de0be0bc100f67b"}
Jul 01 14:39:14.382437 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.382411    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pms96" event={"ID":"93e55270-6d6b-40c6-b27b-20c14c438f0e","Type":"ContainerStarted","Data":"77bdb4bd9d0049198052b5aee4f962180181253a0a791e628babc0da419d7efa"}
Jul 01 14:39:14.384988 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.384963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5cqvx" event={"ID":"9120aa3b-1ffb-484c-9471-601a7b69a1d5","Type":"ContainerStarted","Data":"6b87d1622178c65a45adc397ef8bb1187bf571246a36d046ca654f8d5b6682bf"}
Jul 01 14:39:14.386970 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.386942    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-phzkb" event={"ID":"e0e28ac5-ab10-4108-b51f-185ea52ba4f2","Type":"ContainerStarted","Data":"fe8ec3992e9011da12c1490068de33ddc448b1787cafb46840a1a22830f542ae"}
Jul 01 14:39:14.388494 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.388471    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwvqg" event={"ID":"9abca8ef-87fb-4559-a1ec-f4fd48071abf","Type":"ContainerStarted","Data":"75e94c0fdcac9f96f47a2193dc400a64d00bb414ddbda7eb93230a7f090b0750"}
Jul 01 14:39:14.390027 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.390001    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4lrpd" event={"ID":"7abab760-11aa-4e86-90df-31cbfb591873","Type":"ContainerStarted","Data":"b82d2abe45a0b67b03b1940efc59e04894eb82ed0ec43bce0cfb0b1eea11b71e"}
Jul 01 14:39:14.392312 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.392288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc" event={"ID":"b688282d-3143-4fc3-94ab-30cdc3bc57d9","Type":"ContainerStarted","Data":"179a8cc7a50213fb2279de30eaf76491b2e211c4a015eed7b0d7342415edacd5"}
Jul 01 14:39:14.789715 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:14.789623    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:39:15.010214 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:15.010181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:15.010376 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:15.010343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:15.010497 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.010481    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:15.010557 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.010547    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret podName:3cd6a69f-c352-40c4-a8b9-125145b3ec48 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:17.010529007 +0000 UTC m=+5.071108181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret") pod "global-pull-secret-syncer-t85xw" (UID: "3cd6a69f-c352-40c4-a8b9-125145b3ec48") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:15.010937 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.010917    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:15.011031 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.010974    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs podName:388279eb-7ee2-4130-9f63-3f54e10c7419 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:17.010959769 +0000 UTC m=+5.071538953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs") pod "network-metrics-daemon-khdj5" (UID: "388279eb-7ee2-4130-9f63-3f54e10c7419") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:15.111079 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:15.111000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bnk\" (UniqueName: \"kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk\") pod \"network-check-target-llw6c\" (UID: \"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c\") " pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:15.111255 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.111212    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:39:15.111255 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.111242    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:39:15.111255 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.111255    2558 projected.go:196] Error preparing data for projected volume kube-api-access-l8bnk for pod openshift-network-diagnostics/network-check-target-llw6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:15.111428 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.111311    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk podName:c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c nodeName:}" failed. No retries permitted until 2026-07-01 14:39:17.111293257 +0000 UTC m=+5.171872428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8bnk" (UniqueName: "kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk") pod "network-check-target-llw6c" (UID: "c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:15.325974 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:15.325906    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 14:34:13 +0000 UTC" deadline="2028-03-04 11:12:06.900401183 +0000 UTC"
Jul 01 14:39:15.325974 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:15.325945    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14684h32m51.574461105s"
Jul 01 14:39:15.347864 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:15.347832    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:15.348035 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.347956    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:15.348591 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:15.348241    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:15.348591 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.348356    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:15.348591 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:15.348412    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:15.348591 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:15.348496    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:17.026463 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:17.026381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:17.026463 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:17.026450    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:17.026983 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.026585    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:17.026983 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.026640    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret podName:3cd6a69f-c352-40c4-a8b9-125145b3ec48 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:21.026622879 +0000 UTC m=+9.087202056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret") pod "global-pull-secret-syncer-t85xw" (UID: "3cd6a69f-c352-40c4-a8b9-125145b3ec48") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:17.026983 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.026834    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:17.026983 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.026882    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs podName:388279eb-7ee2-4130-9f63-3f54e10c7419 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:21.026865261 +0000 UTC m=+9.087444435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs") pod "network-metrics-daemon-khdj5" (UID: "388279eb-7ee2-4130-9f63-3f54e10c7419") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:17.127598 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:17.127565    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bnk\" (UniqueName: \"kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk\") pod \"network-check-target-llw6c\" (UID: \"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c\") " pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:17.127807 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.127718    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:39:17.127807 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.127742    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:39:17.127807 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.127756    2558 projected.go:196] Error preparing data for projected volume kube-api-access-l8bnk for pod openshift-network-diagnostics/network-check-target-llw6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:17.127807 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.127808    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk podName:c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c nodeName:}" failed. No retries permitted until 2026-07-01 14:39:21.127790734 +0000 UTC m=+9.188369919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8bnk" (UniqueName: "kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk") pod "network-check-target-llw6c" (UID: "c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:17.349819 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:17.349473    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:17.349819 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.349602    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:17.349819 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:17.349756    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:17.349819 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:17.349769    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:17.350139 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.349864    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:17.350139 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:17.349961    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:19.347871 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:19.347814    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:19.347871 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:19.347848    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:19.348387 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:19.347972    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:19.348387 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:19.348039    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:19.348387 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:19.348099    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:19.348387 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:19.348172    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:21.061202 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:21.060526    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:21.061202 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:21.060588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:21.061202 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.060721    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:21.061202 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.060780    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret podName:3cd6a69f-c352-40c4-a8b9-125145b3ec48 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:29.060763284 +0000 UTC m=+17.121342472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret") pod "global-pull-secret-syncer-t85xw" (UID: "3cd6a69f-c352-40c4-a8b9-125145b3ec48") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:21.061202 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.060847    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:21.061202 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.060879    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs podName:388279eb-7ee2-4130-9f63-3f54e10c7419 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:29.06086688 +0000 UTC m=+17.121446052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs") pod "network-metrics-daemon-khdj5" (UID: "388279eb-7ee2-4130-9f63-3f54e10c7419") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:21.161313 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:21.161196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bnk\" (UniqueName: \"kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk\") pod \"network-check-target-llw6c\" (UID: \"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c\") " pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:21.161507 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.161480    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:39:21.161597 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.161513    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:39:21.161597 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.161528    2558 projected.go:196] Error preparing data for projected volume kube-api-access-l8bnk for pod openshift-network-diagnostics/network-check-target-llw6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:21.161597 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.161591    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk podName:c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c nodeName:}" failed. No retries permitted until 2026-07-01 14:39:29.161572057 +0000 UTC m=+17.222151226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8bnk" (UniqueName: "kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk") pod "network-check-target-llw6c" (UID: "c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:21.348651 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:21.348618    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:21.348651 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:21.348645    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:21.348908 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:21.348618    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:21.348908 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.348762    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:21.348908 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.348877    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:21.349067 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:21.348952    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:23.348769 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:23.348385    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:23.348769 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:23.348505    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:23.348769 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:23.348400    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:23.348769 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:23.348592    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:23.348769 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:23.348390    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:23.348769 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:23.348673    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:25.348260 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:25.348217    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:25.348721 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:25.348231    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:25.348721 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:25.348354    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:25.348721 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:25.348360    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:25.348721 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:25.348443    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:25.348721 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:25.348524    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:27.347703 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:27.347657    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:27.348137 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:27.347657    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:27.348137 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:27.347805    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:27.348137 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:27.347657    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:27.348137 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:27.347862    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:27.348137 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:27.347939    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:29.118723 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:29.118648    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:29.119205 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:29.118734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:29.119205 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.118809    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:29.119205 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.118876    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:29.119205 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.118904    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs podName:388279eb-7ee2-4130-9f63-3f54e10c7419 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:45.118882253 +0000 UTC m=+33.179461442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs") pod "network-metrics-daemon-khdj5" (UID: "388279eb-7ee2-4130-9f63-3f54e10c7419") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:29.119205 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.118928    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret podName:3cd6a69f-c352-40c4-a8b9-125145b3ec48 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:45.118911818 +0000 UTC m=+33.179490990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret") pod "global-pull-secret-syncer-t85xw" (UID: "3cd6a69f-c352-40c4-a8b9-125145b3ec48") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:29.219319 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:29.219274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bnk\" (UniqueName: \"kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk\") pod \"network-check-target-llw6c\" (UID: \"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c\") " pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:29.219507 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.219486    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:39:29.219586 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.219514    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:39:29.219586 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.219529    2558 projected.go:196] Error preparing data for projected volume kube-api-access-l8bnk for pod openshift-network-diagnostics/network-check-target-llw6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:29.219681 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.219600    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk podName:c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c nodeName:}" failed. No retries permitted until 2026-07-01 14:39:45.219580849 +0000 UTC m=+33.280160019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8bnk" (UniqueName: "kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk") pod "network-check-target-llw6c" (UID: "c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:29.348667 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:29.348630    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:29.348864 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:29.348628    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:29.348864 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.348767    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:29.348864 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:29.348772    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:29.349025 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.348874    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:29.349025 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:29.348975    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:31.348409 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:31.348368    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:31.348871 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:31.348369    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:31.348871 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:31.348482    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:31.348871 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:31.348580    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:31.348871 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:31.348369    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:31.348871 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:31.348746    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:33.348325 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:33.348253    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:33.348796 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:33.348392    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:33.348796 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:33.348459    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:33.348796 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:33.348547    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:33.348796 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:33.348591    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:33.348796 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:33.348717    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:33.431292 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:33.431258    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pms96" event={"ID":"93e55270-6d6b-40c6-b27b-20c14c438f0e","Type":"ContainerStarted","Data":"02620543d92b7e8d8574d1a50cb72ef1ca1acba5f3f210f04ddfd31a9527fdc0"}
Jul 01 14:39:33.432708 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:33.432661    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" event={"ID":"1bdd498e17258fb69bcdf5cab2dc7417","Type":"ContainerStarted","Data":"61178042df9445904b5c2cb337382a170ec2541340e7094980ee8e481575edfb"}
Jul 01 14:39:33.433933 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:33.433912    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5cqvx" event={"ID":"9120aa3b-1ffb-484c-9471-601a7b69a1d5","Type":"ContainerStarted","Data":"8f45e9be097e7c69d29b8093a4e89d0e44c16ddda0add1eb97eb11485b1894b4"}
Jul 01 14:39:33.501641 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:33.501589    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pms96" podStartSLOduration=2.185109458 podStartE2EDuration="21.501570223s" totalImagesPullingTime="19.316460765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:33.462948661 +0000 UTC m=+21.523527854" watchObservedRunningTime="2026-07-01 14:39:33.501570223 +0000 UTC m=+21.562149414"
Jul 01 14:39:33.534840 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:33.534788    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-20.ec2.internal" podStartSLOduration=20.534766094 podStartE2EDuration="20.534766094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:33.502402669 +0000 UTC m=+21.562981860" watchObservedRunningTime="2026-07-01 14:39:33.534766094 +0000 UTC m=+21.595345286"
Jul 01 14:39:33.534958 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:33.534925    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5cqvx" podStartSLOduration=2.232263154 podStartE2EDuration="21.534916642s" totalImagesPullingTime="19.302653488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:33.53393387 +0000 UTC m=+21.594513076" watchObservedRunningTime="2026-07-01 14:39:33.534916642 +0000 UTC m=+21.595495834"
Jul 01 14:39:34.437640 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.437413    2558 generic.go:356] "Generic (PLEG): container finished" podID="9abca8ef-87fb-4559-a1ec-f4fd48071abf" containerID="e47bff42a13d8a2d8ef98ea728e2a36702f080012dff02262893d607caa80e5a" exitCode=0
Jul 01 14:39:34.437640 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.437500    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwvqg" event={"ID":"9abca8ef-87fb-4559-a1ec-f4fd48071abf","Type":"ContainerDied","Data":"e47bff42a13d8a2d8ef98ea728e2a36702f080012dff02262893d607caa80e5a"}
Jul 01 14:39:34.439261 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.439237    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4lrpd" event={"ID":"7abab760-11aa-4e86-90df-31cbfb591873","Type":"ContainerStarted","Data":"4c87f020f44654087f9bb1c8766041fcf06b3073686fb17eb9fa629437a6e53b"}
Jul 01 14:39:34.440770 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.440678    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc" event={"ID":"b688282d-3143-4fc3-94ab-30cdc3bc57d9","Type":"ContainerStarted","Data":"cdbdf11fc6a4f01f8d5e9527d201e1848ee08284a83b2f2387ca75b379537cef"}
Jul 01 14:39:34.442019 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.441998    2558 generic.go:356] "Generic (PLEG): container finished" podID="a5389ab3ca8a898983720c30b1081103" containerID="54e2931a18e78044914e905b9b1f5658a15c4a946299057f30269c3a32495f08" exitCode=0
Jul 01 14:39:34.442111 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.442066    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" event={"ID":"a5389ab3ca8a898983720c30b1081103","Type":"ContainerDied","Data":"54e2931a18e78044914e905b9b1f5658a15c4a946299057f30269c3a32495f08"}
Jul 01 14:39:34.443512 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.443484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-txzp9" event={"ID":"35f35078-e173-4181-9dbc-ade18144ebd6","Type":"ContainerStarted","Data":"b3dd55bf9df2b31ffe80162dfa90aa5f55b6b206192b08dc1a5dbaf7162b3884"}
Jul 01 14:39:34.446203 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.446187    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:39:34.446513 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.446496    2558 generic.go:356] "Generic (PLEG): container finished" podID="95dbc66a-5975-4106-ba5d-9564565d86f2" containerID="6f776696cb68b5291745fdd211a6f2b87ef64322d21d26c969593770b96e4ae2" exitCode=1
Jul 01 14:39:34.446597 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.446564    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" event={"ID":"95dbc66a-5975-4106-ba5d-9564565d86f2","Type":"ContainerStarted","Data":"43ba3781a9b1e6b547b66be2e836461ff43489598ee3d576494e0264fd22e816"}
Jul 01 14:39:34.446665 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.446649    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" event={"ID":"95dbc66a-5975-4106-ba5d-9564565d86f2","Type":"ContainerStarted","Data":"2a60dce3224563785afb8d0ef401660c6ae0ad8d86b7dd48a239623f86ffe2cd"}
Jul 01 14:39:34.446736 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.446707    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" event={"ID":"95dbc66a-5975-4106-ba5d-9564565d86f2","Type":"ContainerStarted","Data":"2eea2cc9c1b6f35dff7526e72caafe3df61e1847157c48d59a94897f5bd8d627"}
Jul 01 14:39:34.446782 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.446740    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" event={"ID":"95dbc66a-5975-4106-ba5d-9564565d86f2","Type":"ContainerStarted","Data":"aecd918b78143b40305229dac849517a31ba34c8939bd732f678acd8a665915a"}
Jul 01 14:39:34.446828 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.446780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" event={"ID":"95dbc66a-5975-4106-ba5d-9564565d86f2","Type":"ContainerDied","Data":"6f776696cb68b5291745fdd211a6f2b87ef64322d21d26c969593770b96e4ae2"}
Jul 01 14:39:34.446874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.446828    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" event={"ID":"95dbc66a-5975-4106-ba5d-9564565d86f2","Type":"ContainerStarted","Data":"16a9c3ac5664e2bc92daa6d42bdd11478cd7112ef7534f90f78894795ff1a6ba"}
Jul 01 14:39:34.447886 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.447862    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztcz2" event={"ID":"0b05de0b-5d43-4566-b311-699a60bbd0cf","Type":"ContainerStarted","Data":"109b9d04aad78e886ce140d611424bf3b94b9254b0ea41cfecb2d5282284f1f7"}
Jul 01 14:39:34.449092 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.449071    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-phzkb" event={"ID":"e0e28ac5-ab10-4108-b51f-185ea52ba4f2","Type":"ContainerStarted","Data":"b7a49ca28254410f82c8162f120283bfdb2488cafb68d34e1c0dee31d5e1aeeb"}
Jul 01 14:39:34.480673 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.480496    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ztcz2" podStartSLOduration=7.589230128 podStartE2EDuration="22.480482558s" totalImagesPullingTime="14.89125243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:34.480380579 +0000 UTC m=+22.540959773" watchObservedRunningTime="2026-07-01 14:39:34.480482558 +0000 UTC m=+22.541061751"
Jul 01 14:39:34.498560 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.498517    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-phzkb" podStartSLOduration=7.629279532 podStartE2EDuration="22.49849894s" totalImagesPullingTime="14.869219408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:34.49837502 +0000 UTC m=+22.558954211" watchObservedRunningTime="2026-07-01 14:39:34.49849894 +0000 UTC m=+22.559078133"
Jul 01 14:39:34.518442 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.518408    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-txzp9" podStartSLOduration=3.491046603 podStartE2EDuration="22.518395517s" totalImagesPullingTime="19.027348914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:34.518025732 +0000 UTC m=+22.578604923" watchObservedRunningTime="2026-07-01 14:39:34.518395517 +0000 UTC m=+22.578974708"
Jul 01 14:39:34.565923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.565882    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4lrpd" podStartSLOduration=3.221220926 podStartE2EDuration="22.565868174s" totalImagesPullingTime="19.344647248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:34.565491838 +0000 UTC m=+22.626071029" watchObservedRunningTime="2026-07-01 14:39:34.565868174 +0000 UTC m=+22.626447365"
Jul 01 14:39:34.616091 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:34.616061    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:35.348495 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:35.348465    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:35.348746 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:35.348554    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:35.348746 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:35.348557    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:35.348746 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:35.348599    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:35.348746 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:35.348710    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:35.348905 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:35.348776    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:35.452357 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:35.452323    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" event={"ID":"a5389ab3ca8a898983720c30b1081103","Type":"ContainerStarted","Data":"7537bfec3893c419408499450d98efe6e95dd4b1431372f8ee2aa6afbbf05eba"}
Jul 01 14:39:35.473680 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:35.473634    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-20.ec2.internal" podStartSLOduration=22.47361927 podStartE2EDuration="22.47361927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:35.473442463 +0000 UTC m=+23.534021645" watchObservedRunningTime="2026-07-01 14:39:35.47361927 +0000 UTC m=+23.534198460"
Jul 01 14:39:35.675723 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:35.675697    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 14:39:36.188099 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:36.188071    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:36.188713 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:36.188671    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:36.364881 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:36.364758    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T14:39:35.675714037Z","UUID":"1e81d60d-a29d-4c37-b12e-5a328ad8cb4c","Handler":null,"Name":"","Endpoint":""}
Jul 01 14:39:36.366406 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:36.366379    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 14:39:36.366519 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:36.366426    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 14:39:36.456465 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:36.456376    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc" event={"ID":"b688282d-3143-4fc3-94ab-30cdc3bc57d9","Type":"ContainerStarted","Data":"f5aa88302f7d88e6bf8c9bf56bb7503b228f7e387ec5ed19c933434afa5cec4e"}
Jul 01 14:39:36.460020 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:36.459997    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:39:36.460729 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:36.460703    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" event={"ID":"95dbc66a-5975-4106-ba5d-9564565d86f2","Type":"ContainerStarted","Data":"9026668ae801f5293d4cda7a0bae8576d63c91918e49ab471530087e291ada75"}
Jul 01 14:39:36.461674 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:36.461654    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-phzkb"
Jul 01 14:39:37.348084 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:37.348048    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:37.348267 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:37.348090    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:37.348267 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:37.348090    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:37.348267 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:37.348192    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:37.348405 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:37.348262    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:37.348405 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:37.348350    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:37.465372 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:37.465342    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc" event={"ID":"b688282d-3143-4fc3-94ab-30cdc3bc57d9","Type":"ContainerStarted","Data":"0dff1086c8d30158ab47474c455410261ffc40ad149d4133bc020a6667fdf10f"}
Jul 01 14:39:38.472102 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:38.471898    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:39:38.472918 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:38.472850    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" event={"ID":"95dbc66a-5975-4106-ba5d-9564565d86f2","Type":"ContainerStarted","Data":"c18383e7c06159535ca08b5a17a1bf1056e0ade1afbfb7702cbd3856c8302106"}
Jul 01 14:39:39.348168 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:39.348135    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:39.348371 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:39.348251    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:39.348371 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:39.348264    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:39.348371 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:39.348288    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:39.348493 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:39.348403    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:39.348537 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:39.348491    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:39.475086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:39.475055    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:39.475650 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:39.475246    2558 scope.go:117] "RemoveContainer" containerID="6f776696cb68b5291745fdd211a6f2b87ef64322d21d26c969593770b96e4ae2"
Jul 01 14:39:39.491742 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:39.491717    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:39.506276 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:39.506228    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6lzxc" podStartSLOduration=5.688525259 podStartE2EDuration="27.506213635s" totalImagesPullingTime="21.817688376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:37.486276684 +0000 UTC m=+25.546855898" watchObservedRunningTime="2026-07-01 14:39:39.506213635 +0000 UTC m=+27.566792826"
Jul 01 14:39:40.480119 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:40.480100    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:39:40.480490 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:40.480408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" event={"ID":"95dbc66a-5975-4106-ba5d-9564565d86f2","Type":"ContainerStarted","Data":"5f8dd527224f31647f9d219b89af91dfd7f136160e56b0da1c5f988df014118c"}
Jul 01 14:39:40.480567 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:40.480551    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 01 14:39:40.480818 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:40.480796    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:40.498739 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:40.498656    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:40.517765 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:40.517710    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" podStartSLOduration=8.650251431000001 podStartE2EDuration="28.517668945s" totalImagesPullingTime="19.867417514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:40.516347713 +0000 UTC m=+28.576926907" watchObservedRunningTime="2026-07-01 14:39:40.517668945 +0000 UTC m=+28.578248136"
Jul 01 14:39:41.348603 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:41.348574    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:41.348871 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:41.348577    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:41.348871 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:41.348661    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:41.348871 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:41.348705    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:41.348871 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:41.348790    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:41.348871 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:41.348846    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:41.484273 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:41.484238    2558 generic.go:356] "Generic (PLEG): container finished" podID="9abca8ef-87fb-4559-a1ec-f4fd48071abf" containerID="b080c7302d132298d03932f670454f613314d64aad440f7cea81c1962670d7c7" exitCode=0
Jul 01 14:39:41.484833 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:41.484319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwvqg" event={"ID":"9abca8ef-87fb-4559-a1ec-f4fd48071abf","Type":"ContainerDied","Data":"b080c7302d132298d03932f670454f613314d64aad440f7cea81c1962670d7c7"}
Jul 01 14:39:41.484833 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:41.484396    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 01 14:39:41.496308 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:41.496289    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:39:42.488757 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:42.488648    2558 generic.go:356] "Generic (PLEG): container finished" podID="9abca8ef-87fb-4559-a1ec-f4fd48071abf" containerID="98c0a360f2a0289fc56963c62e124d27247fce595738cc0d3e960d4a2b9a4b4d" exitCode=0
Jul 01 14:39:42.488757 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:42.488729    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwvqg" event={"ID":"9abca8ef-87fb-4559-a1ec-f4fd48071abf","Type":"ContainerDied","Data":"98c0a360f2a0289fc56963c62e124d27247fce595738cc0d3e960d4a2b9a4b4d"}
Jul 01 14:39:43.347986 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:43.347955    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:43.348137 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:43.348048    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:43.348137 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:43.348057    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:43.348137 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:43.348054    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:43.348244 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:43.348139    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:43.348244 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:43.348196    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:43.493410 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:43.493332    2558 generic.go:356] "Generic (PLEG): container finished" podID="9abca8ef-87fb-4559-a1ec-f4fd48071abf" containerID="d14e28a1200636c0a140d337e56c95458f14b6f7fd3ba1ea52d06ccd1beeb732" exitCode=0
Jul 01 14:39:43.493786 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:43.493409    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwvqg" event={"ID":"9abca8ef-87fb-4559-a1ec-f4fd48071abf","Type":"ContainerDied","Data":"d14e28a1200636c0a140d337e56c95458f14b6f7fd3ba1ea52d06ccd1beeb732"}
Jul 01 14:39:43.503597 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:43.503568    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp" podUID="95dbc66a-5975-4106-ba5d-9564565d86f2" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 01 14:39:45.141984 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:45.141955    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:45.142418 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:45.142002    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:45.142418 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.142117    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:45.142418 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.142120    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:45.142418 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.142179    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret podName:3cd6a69f-c352-40c4-a8b9-125145b3ec48 nodeName:}" failed. No retries permitted until 2026-07-01 14:40:17.142162191 +0000 UTC m=+65.202741363 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret") pod "global-pull-secret-syncer-t85xw" (UID: "3cd6a69f-c352-40c4-a8b9-125145b3ec48") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:45.142418 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.142195    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs podName:388279eb-7ee2-4130-9f63-3f54e10c7419 nodeName:}" failed. No retries permitted until 2026-07-01 14:40:17.142187642 +0000 UTC m=+65.202766810 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs") pod "network-metrics-daemon-khdj5" (UID: "388279eb-7ee2-4130-9f63-3f54e10c7419") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:45.243081 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:45.243040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bnk\" (UniqueName: \"kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk\") pod \"network-check-target-llw6c\" (UID: \"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c\") " pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:45.243253 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.243209    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:39:45.243253 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.243235    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:39:45.243253 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.243250    2558 projected.go:196] Error preparing data for projected volume kube-api-access-l8bnk for pod openshift-network-diagnostics/network-check-target-llw6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:45.243430 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.243309    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk podName:c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c nodeName:}" failed. No retries permitted until 2026-07-01 14:40:17.243290373 +0000 UTC m=+65.303869559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8bnk" (UniqueName: "kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk") pod "network-check-target-llw6c" (UID: "c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:45.348442 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:45.348404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:45.348442 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:45.348439    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:45.348666 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:45.348404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:45.348666 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.348531    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:45.348666 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.348635    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:45.348853 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:45.348727    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:47.348398 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:47.348362    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:47.348398 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:47.348402    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:47.348943 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:47.348401    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:47.348943 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:47.348489    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:47.348943 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:47.348567    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:47.348943 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:47.348654    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:48.508158 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:48.508060    2558 generic.go:356] "Generic (PLEG): container finished" podID="9abca8ef-87fb-4559-a1ec-f4fd48071abf" containerID="f57d7312c8b501aba39df8f283c19f9ea604d1092b438fc966a7790071067544" exitCode=0
Jul 01 14:39:48.508158 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:48.508114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwvqg" event={"ID":"9abca8ef-87fb-4559-a1ec-f4fd48071abf","Type":"ContainerDied","Data":"f57d7312c8b501aba39df8f283c19f9ea604d1092b438fc966a7790071067544"}
Jul 01 14:39:49.348288 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:49.348256    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:49.348288 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:49.348289    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:49.348519 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:49.348256    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:49.348519 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:49.348359    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:49.348519 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:49.348431    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:49.348519 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:49.348488    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:49.513396 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:49.513364    2558 generic.go:356] "Generic (PLEG): container finished" podID="9abca8ef-87fb-4559-a1ec-f4fd48071abf" containerID="cbb88f1fa942c4a40881c4b651e6c97b291669a1f31524a9a0159dc780c7995b" exitCode=0
Jul 01 14:39:49.513810 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:49.513428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwvqg" event={"ID":"9abca8ef-87fb-4559-a1ec-f4fd48071abf","Type":"ContainerDied","Data":"cbb88f1fa942c4a40881c4b651e6c97b291669a1f31524a9a0159dc780c7995b"}
Jul 01 14:39:50.518446 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:50.518416    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwvqg" event={"ID":"9abca8ef-87fb-4559-a1ec-f4fd48071abf","Type":"ContainerStarted","Data":"7c208925286b1533e1d0fb774d80b0dec943d7bd13f4d655e0fe1246b8be5a1a"}
Jul 01 14:39:50.548084 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:50.548041    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bwvqg" podStartSLOduration=7.207336778 podStartE2EDuration="38.548026952s" totalImagesPullingTime="31.340690174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:50.547082161 +0000 UTC m=+38.607661353" watchObservedRunningTime="2026-07-01 14:39:50.548026952 +0000 UTC m=+38.608606143"
Jul 01 14:39:51.348171 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:51.348137    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:51.348171 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:51.348169    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:51.348382 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:51.348225    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:51.348382 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:51.348235    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:51.348382 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:51.348290    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:51.348382 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:51.348370    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:53.348566 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:53.348534    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:53.348964 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:53.348625    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:53.348964 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:53.348538    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:53.348964 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:53.348676    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:53.348964 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:53.348772    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:53.348964 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:53.348815    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:55.348433 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:55.348401    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:55.348804 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:55.348494    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:55.348804 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:55.348503    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:55.348804 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:55.348570    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:55.348804 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:55.348582    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:55.348804 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:55.348672    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:57.348415 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:57.348386    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:57.348801 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:57.348416    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:57.348801 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:57.348420    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:57.348801 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:57.348490    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:57.348801 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:57.348586    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:57.348801 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:57.348656    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:57.664710 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:57.664620    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t85xw"]
Jul 01 14:39:57.664839 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:57.664755    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:57.664874 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:57.664839    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:39:57.667827 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:57.667804    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-llw6c"]
Jul 01 14:39:57.667976 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:57.667909    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:57.668058 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:57.668037    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:57.668750 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:57.668727    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-khdj5"]
Jul 01 14:39:57.668864 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:57.668813    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:57.668932 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:57.668909    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:59.347977 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:59.347943    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:39:59.348384 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:59.348042    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:39:59.348384 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:59.348046    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:39:59.348384 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:39:59.348052    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:39:59.348384 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:59.348124    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:39:59.348384 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:39:59.348176    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:40:01.348373 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:01.348200    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:40:01.348766 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:01.348199    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:40:01.348766 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:40:01.348453    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t85xw" podUID="3cd6a69f-c352-40c4-a8b9-125145b3ec48"
Jul 01 14:40:01.348766 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:40:01.348503    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llw6c" podUID="c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c"
Jul 01 14:40:01.348766 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:01.348199    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:40:01.348766 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:40:01.348600    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khdj5" podUID="388279eb-7ee2-4130-9f63-3f54e10c7419"
Jul 01 14:40:02.222607 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.222544    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-20.ec2.internal" event="NodeReady"
Jul 01 14:40:02.222755 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.222663    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 14:40:02.310571 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.310537    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l8l96"]
Jul 01 14:40:02.325870 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.325850    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9gwv6"]
Jul 01 14:40:02.325989 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.325972    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.329577 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.329556    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 14:40:02.329710 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.329698    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rvr9p\""
Jul 01 14:40:02.329951 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.329934    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 14:40:02.339956 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.339942    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9gwv6"
Jul 01 14:40:02.343070 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.343052    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 14:40:02.343172 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.343158    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tggwd\""
Jul 01 14:40:02.343237 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.343211    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 14:40:02.343478 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.343464    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 14:40:02.345147 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.345121    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l8l96"]
Jul 01 14:40:02.347037 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.347018    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9gwv6"]
Jul 01 14:40:02.375453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.375431    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec001fb-e32a-4a0c-a80e-539e1f53f108-metrics-tls\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.375853 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.375469    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec001fb-e32a-4a0c-a80e-539e1f53f108-tmp-dir\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.375853 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.375488    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsfqz\" (UniqueName: \"kubernetes.io/projected/6ec001fb-e32a-4a0c-a80e-539e1f53f108-kube-api-access-dsfqz\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.375853 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.375537    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec001fb-e32a-4a0c-a80e-539e1f53f108-config-volume\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.401777 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.401754    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vwk6n"]
Jul 01 14:40:02.424341 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.424316    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vwk6n"]
Jul 01 14:40:02.426185 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.424471    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.427884 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.427862    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 14:40:02.427884 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.427880    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 14:40:02.428119 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.428099    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 14:40:02.428217 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.428188    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 14:40:02.428642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.428626    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nq9qs\""
Jul 01 14:40:02.476446 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476377    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13243029-0c8d-490c-a21b-437e7ca1b85d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.476446 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476415    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec001fb-e32a-4a0c-a80e-539e1f53f108-tmp-dir\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.476446 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476438    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13243029-0c8d-490c-a21b-437e7ca1b85d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.476717 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476501    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsfqz\" (UniqueName: \"kubernetes.io/projected/6ec001fb-e32a-4a0c-a80e-539e1f53f108-kube-api-access-dsfqz\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.476717 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec001fb-e32a-4a0c-a80e-539e1f53f108-config-volume\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.476717 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhjsm\" (UniqueName: \"kubernetes.io/projected/881f795a-433f-43a3-8331-2b2ec04f2d83-kube-api-access-qhjsm\") pod \"ingress-canary-9gwv6\" (UID: \"881f795a-433f-43a3-8331-2b2ec04f2d83\") " pod="openshift-ingress-canary/ingress-canary-9gwv6"
Jul 01 14:40:02.476867 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclrq\" (UniqueName: \"kubernetes.io/projected/13243029-0c8d-490c-a21b-437e7ca1b85d-kube-api-access-cclrq\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.476867 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476776    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec001fb-e32a-4a0c-a80e-539e1f53f108-tmp-dir\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.476867 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476803    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec001fb-e32a-4a0c-a80e-539e1f53f108-metrics-tls\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.476867 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476830    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/881f795a-433f-43a3-8331-2b2ec04f2d83-cert\") pod \"ingress-canary-9gwv6\" (UID: \"881f795a-433f-43a3-8331-2b2ec04f2d83\") " pod="openshift-ingress-canary/ingress-canary-9gwv6"
Jul 01 14:40:02.476867 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13243029-0c8d-490c-a21b-437e7ca1b85d-crio-socket\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.477034 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.476880    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13243029-0c8d-490c-a21b-437e7ca1b85d-data-volume\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.477106 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.477090    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec001fb-e32a-4a0c-a80e-539e1f53f108-config-volume\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.480552 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.480522    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ec001fb-e32a-4a0c-a80e-539e1f53f108-metrics-tls\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.488919 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.488900    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsfqz\" (UniqueName: \"kubernetes.io/projected/6ec001fb-e32a-4a0c-a80e-539e1f53f108-kube-api-access-dsfqz\") pod \"dns-default-l8l96\" (UID: \"6ec001fb-e32a-4a0c-a80e-539e1f53f108\") " pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.577545 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.577510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13243029-0c8d-490c-a21b-437e7ca1b85d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.577545 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.577548    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13243029-0c8d-490c-a21b-437e7ca1b85d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.577840 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.577576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhjsm\" (UniqueName: \"kubernetes.io/projected/881f795a-433f-43a3-8331-2b2ec04f2d83-kube-api-access-qhjsm\") pod \"ingress-canary-9gwv6\" (UID: \"881f795a-433f-43a3-8331-2b2ec04f2d83\") " pod="openshift-ingress-canary/ingress-canary-9gwv6"
Jul 01 14:40:02.577840 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.577608    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclrq\" (UniqueName: \"kubernetes.io/projected/13243029-0c8d-490c-a21b-437e7ca1b85d-kube-api-access-cclrq\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.577840 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.577767    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/881f795a-433f-43a3-8331-2b2ec04f2d83-cert\") pod \"ingress-canary-9gwv6\" (UID: \"881f795a-433f-43a3-8331-2b2ec04f2d83\") " pod="openshift-ingress-canary/ingress-canary-9gwv6"
Jul 01 14:40:02.577840 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.577804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13243029-0c8d-490c-a21b-437e7ca1b85d-crio-socket\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.577840 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.577827    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13243029-0c8d-490c-a21b-437e7ca1b85d-data-volume\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.578116 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.577979    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13243029-0c8d-490c-a21b-437e7ca1b85d-crio-socket\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.578116 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.578093    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13243029-0c8d-490c-a21b-437e7ca1b85d-data-volume\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.578239 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.578215    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13243029-0c8d-490c-a21b-437e7ca1b85d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.579977 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.579956    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13243029-0c8d-490c-a21b-437e7ca1b85d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.580146 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.580130    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/881f795a-433f-43a3-8331-2b2ec04f2d83-cert\") pod \"ingress-canary-9gwv6\" (UID: \"881f795a-433f-43a3-8331-2b2ec04f2d83\") " pod="openshift-ingress-canary/ingress-canary-9gwv6"
Jul 01 14:40:02.586598 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.586578    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhjsm\" (UniqueName: \"kubernetes.io/projected/881f795a-433f-43a3-8331-2b2ec04f2d83-kube-api-access-qhjsm\") pod \"ingress-canary-9gwv6\" (UID: \"881f795a-433f-43a3-8331-2b2ec04f2d83\") " pod="openshift-ingress-canary/ingress-canary-9gwv6"
Jul 01 14:40:02.586895 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.586880    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclrq\" (UniqueName: \"kubernetes.io/projected/13243029-0c8d-490c-a21b-437e7ca1b85d-kube-api-access-cclrq\") pod \"insights-runtime-extractor-vwk6n\" (UID: \"13243029-0c8d-490c-a21b-437e7ca1b85d\") " pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.635814 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.635793    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:02.648925 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.648906    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9gwv6"
Jul 01 14:40:02.735794 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.735700    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vwk6n"
Jul 01 14:40:02.857887 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.857855    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l8l96"]
Jul 01 14:40:02.860628 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.860604    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9gwv6"]
Jul 01 14:40:02.862203 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:02.862182    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec001fb_e32a_4a0c_a80e_539e1f53f108.slice/crio-e5536ab4a184f97c6985020a5e73154d4b0a5a71e9d57fac171ba6b4d9cd84dd WatchSource:0}: Error finding container e5536ab4a184f97c6985020a5e73154d4b0a5a71e9d57fac171ba6b4d9cd84dd: Status 404 returned error can't find the container with id e5536ab4a184f97c6985020a5e73154d4b0a5a71e9d57fac171ba6b4d9cd84dd
Jul 01 14:40:02.863372 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:02.863346    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881f795a_433f_43a3_8331_2b2ec04f2d83.slice/crio-a3b824ce33fa3943594bc284c6f4708197f826549e4a4bb73148795a2c0ba5f3 WatchSource:0}: Error finding container a3b824ce33fa3943594bc284c6f4708197f826549e4a4bb73148795a2c0ba5f3: Status 404 returned error can't find the container with id a3b824ce33fa3943594bc284c6f4708197f826549e4a4bb73148795a2c0ba5f3
Jul 01 14:40:02.889017 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:02.888993    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vwk6n"]
Jul 01 14:40:02.892255 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:02.892232    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13243029_0c8d_490c_a21b_437e7ca1b85d.slice/crio-73fb2458f1bac8102236c76f28ac197b820ed434239327479a2d965466f67a29 WatchSource:0}: Error finding container 73fb2458f1bac8102236c76f28ac197b820ed434239327479a2d965466f67a29: Status 404 returned error can't find the container with id 73fb2458f1bac8102236c76f28ac197b820ed434239327479a2d965466f67a29
Jul 01 14:40:03.232840 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.232808    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"]
Jul 01 14:40:03.250083 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.250050    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.251352 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.251137    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"]
Jul 01 14:40:03.253781 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.253757    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cdjd4\""
Jul 01 14:40:03.253781 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.253771    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 01 14:40:03.253964 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.253802    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 01 14:40:03.253964 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.253946    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 14:40:03.254386 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.254147    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 14:40:03.254386 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.254213    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 14:40:03.283672 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.283641    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ac0443-bb91-4f99-b625-cc0871bd2723-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.283821 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.283678    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhnr\" (UniqueName: \"kubernetes.io/projected/e0ac0443-bb91-4f99-b625-cc0871bd2723-kube-api-access-wlhnr\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.283821 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.283734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0ac0443-bb91-4f99-b625-cc0871bd2723-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.283821 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.283799    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0ac0443-bb91-4f99-b625-cc0871bd2723-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.348512 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.348485    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:40:03.348644 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.348568    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:40:03.348715 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.348679    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:40:03.354674 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.354603    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5778q\""
Jul 01 14:40:03.354800 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.354681    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hvdsl\""
Jul 01 14:40:03.354866 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.354839    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 14:40:03.355262 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.355250    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 14:40:03.355716 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.355702    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 14:40:03.357074 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.357057    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9f9dbd8fb-9lmcf"]
Jul 01 14:40:03.357550 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.357535    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 14:40:03.368237 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.368222    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.371764 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.370766    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bxph6\""
Jul 01 14:40:03.371764 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.371552    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 14:40:03.372545 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.372319    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 14:40:03.372545 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.372418    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 14:40:03.372976 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.372957    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 14:40:03.373911 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.373884    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 14:40:03.374058 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.374041    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 14:40:03.374435 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.374421    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 14:40:03.379848 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.379823    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9f9dbd8fb-9lmcf"]
Jul 01 14:40:03.383901 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.383880    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 01 14:40:03.384114 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-serving-cert\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.384188 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ac0443-bb91-4f99-b625-cc0871bd2723-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.384240 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhnr\" (UniqueName: \"kubernetes.io/projected/e0ac0443-bb91-4f99-b625-cc0871bd2723-kube-api-access-wlhnr\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.384240 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-oauth-config\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.384346 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384245    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-trusted-ca-bundle\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.384346 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0ac0443-bb91-4f99-b625-cc0871bd2723-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.384439 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384349    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-console-config\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.384439 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-service-ca\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.384439 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384407    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-oauth-serving-cert\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.384608 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384472    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0ac0443-bb91-4f99-b625-cc0871bd2723-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.384608 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.384493    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw45g\" (UniqueName: \"kubernetes.io/projected/97e72214-c19b-401e-a6ed-f807439a575a-kube-api-access-hw45g\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.385308 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.385267    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0ac0443-bb91-4f99-b625-cc0871bd2723-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.387300 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.387278    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0ac0443-bb91-4f99-b625-cc0871bd2723-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.387419 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.387399    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ac0443-bb91-4f99-b625-cc0871bd2723-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.421304 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.421282    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhnr\" (UniqueName: \"kubernetes.io/projected/e0ac0443-bb91-4f99-b625-cc0871bd2723-kube-api-access-wlhnr\") pod \"prometheus-operator-7b7d896497-8jvxk\" (UID: \"e0ac0443-bb91-4f99-b625-cc0871bd2723\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.485405 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.485319    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw45g\" (UniqueName: \"kubernetes.io/projected/97e72214-c19b-401e-a6ed-f807439a575a-kube-api-access-hw45g\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.485405 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.485355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-serving-cert\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.485405 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.485402    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-oauth-config\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.485680 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.485427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-trusted-ca-bundle\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.485755 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.485681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-console-config\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.485755 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.485737    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-service-ca\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.485858 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.485760    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-oauth-serving-cert\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.486471 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.486443    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-oauth-serving-cert\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.486584 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.486484    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-service-ca\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.486584 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.486530    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-trusted-ca-bundle\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.488013 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.487980    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-console-config\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.488533 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.488510    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-serving-cert\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.488699 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.488664    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-oauth-config\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.498480 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.498457    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw45g\" (UniqueName: \"kubernetes.io/projected/97e72214-c19b-401e-a6ed-f807439a575a-kube-api-access-hw45g\") pod \"console-9f9dbd8fb-9lmcf\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") " pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.547638 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.547599    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l8l96" event={"ID":"6ec001fb-e32a-4a0c-a80e-539e1f53f108","Type":"ContainerStarted","Data":"e5536ab4a184f97c6985020a5e73154d4b0a5a71e9d57fac171ba6b4d9cd84dd"}
Jul 01 14:40:03.549217 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.549174    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vwk6n" event={"ID":"13243029-0c8d-490c-a21b-437e7ca1b85d","Type":"ContainerStarted","Data":"73fb2458f1bac8102236c76f28ac197b820ed434239327479a2d965466f67a29"}
Jul 01 14:40:03.550436 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.550407    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9gwv6" event={"ID":"881f795a-433f-43a3-8331-2b2ec04f2d83","Type":"ContainerStarted","Data":"a3b824ce33fa3943594bc284c6f4708197f826549e4a4bb73148795a2c0ba5f3"}
Jul 01 14:40:03.560671 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.560644    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"
Jul 01 14:40:03.681391 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.681141    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:03.749299 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.749154    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-8jvxk"]
Jul 01 14:40:03.752222 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:03.752188    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ac0443_bb91_4f99_b625_cc0871bd2723.slice/crio-b32e899784df48322a313183a5453c940b7e991e4955cac13e6207bbdc164a68 WatchSource:0}: Error finding container b32e899784df48322a313183a5453c940b7e991e4955cac13e6207bbdc164a68: Status 404 returned error can't find the container with id b32e899784df48322a313183a5453c940b7e991e4955cac13e6207bbdc164a68
Jul 01 14:40:03.898818 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:03.898568    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9f9dbd8fb-9lmcf"]
Jul 01 14:40:03.903261 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:03.903232    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e72214_c19b_401e_a6ed_f807439a575a.slice/crio-685c8b3b5bc9c231055fd33e7ec716fb746cca55a3375d5274275db6549b48bc WatchSource:0}: Error finding container 685c8b3b5bc9c231055fd33e7ec716fb746cca55a3375d5274275db6549b48bc: Status 404 returned error can't find the container with id 685c8b3b5bc9c231055fd33e7ec716fb746cca55a3375d5274275db6549b48bc
Jul 01 14:40:04.556479 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:04.556356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f9dbd8fb-9lmcf" event={"ID":"97e72214-c19b-401e-a6ed-f807439a575a","Type":"ContainerStarted","Data":"685c8b3b5bc9c231055fd33e7ec716fb746cca55a3375d5274275db6549b48bc"}
Jul 01 14:40:04.558334 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:04.558240    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk" event={"ID":"e0ac0443-bb91-4f99-b625-cc0871bd2723","Type":"ContainerStarted","Data":"b32e899784df48322a313183a5453c940b7e991e4955cac13e6207bbdc164a68"}
Jul 01 14:40:10.575758 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.575725    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vwk6n" event={"ID":"13243029-0c8d-490c-a21b-437e7ca1b85d","Type":"ContainerStarted","Data":"436fa0f711b159ad5838a06794bb66e2c54f35683cb3bc3229fce16e8acd6c9e"}
Jul 01 14:40:10.577115 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.577095    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9gwv6" event={"ID":"881f795a-433f-43a3-8331-2b2ec04f2d83","Type":"ContainerStarted","Data":"de20e0e9d1e2542c1bfa1aa0c1b0a5937fc39271581fddb01d25142db007b891"}
Jul 01 14:40:10.578566 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.578544    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f9dbd8fb-9lmcf" event={"ID":"97e72214-c19b-401e-a6ed-f807439a575a","Type":"ContainerStarted","Data":"61c0c4a3fbad4b2a0e5c7e4a90fb27ce06f56160b42754437c1f86210abdee54"}
Jul 01 14:40:10.580249 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.580225    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk" event={"ID":"e0ac0443-bb91-4f99-b625-cc0871bd2723","Type":"ContainerStarted","Data":"d7064ef502af1788ac7b65d68ea6b697980ccbfa7d6cf4df6efd42b34293fd33"}
Jul 01 14:40:10.580357 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.580271    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk" event={"ID":"e0ac0443-bb91-4f99-b625-cc0871bd2723","Type":"ContainerStarted","Data":"f95b84faacc515b120a739da85f227a9ec3a3c98ca9948813e1e19ae49b57685"}
Jul 01 14:40:10.581885 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.581868    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l8l96" event={"ID":"6ec001fb-e32a-4a0c-a80e-539e1f53f108","Type":"ContainerStarted","Data":"0c2758fad92047ba5f1cad259a9178843b56141298965a1366f1fba9ec46d90a"}
Jul 01 14:40:10.581961 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.581902    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l8l96" event={"ID":"6ec001fb-e32a-4a0c-a80e-539e1f53f108","Type":"ContainerStarted","Data":"00ed2b0d7a814321c2d2b7c2158b6b19dc15b72c814850f3bb4b9e66888d0c4c"}
Jul 01 14:40:10.582008 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.581988    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:10.596551 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.596453    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9gwv6" podStartSLOduration=1.888925752 podStartE2EDuration="8.596431775s" totalImagesPullingTime="6.707506023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:10.595989027 +0000 UTC m=+58.656568218" watchObservedRunningTime="2026-07-01 14:40:10.596431775 +0000 UTC m=+58.657011025"
Jul 01 14:40:10.614940 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.614893    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l8l96" podStartSLOduration=3.286459718 podStartE2EDuration="8.614874834s" totalImagesPullingTime="5.328415116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:10.614211983 +0000 UTC m=+58.674791175" watchObservedRunningTime="2026-07-01 14:40:10.614874834 +0000 UTC m=+58.675454028"
Jul 01 14:40:10.634410 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.634364    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-8jvxk" podStartSLOduration=1.782204053 podStartE2EDuration="7.634345383s" totalImagesPullingTime="5.85214133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:10.63323985 +0000 UTC m=+58.693819100" watchObservedRunningTime="2026-07-01 14:40:10.634345383 +0000 UTC m=+58.694924577"
Jul 01 14:40:10.653453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:10.653392    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9f9dbd8fb-9lmcf" podStartSLOduration=1.94590167 podStartE2EDuration="7.653372011s" totalImagesPullingTime="5.707470341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:10.652436105 +0000 UTC m=+58.713015295" watchObservedRunningTime="2026-07-01 14:40:10.653372011 +0000 UTC m=+58.713951207"
Jul 01 14:40:11.587288 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:11.587244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vwk6n" event={"ID":"13243029-0c8d-490c-a21b-437e7ca1b85d","Type":"ContainerStarted","Data":"7939b02f79f2a51bdb26cb614246a2293b5396604242c05a973e7ab2cf86e38b"}
Jul 01 14:40:12.601843 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.601811    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"]
Jul 01 14:40:12.604960 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.604939    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.607632 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.607611    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 01 14:40:12.607765 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.607664    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 01 14:40:12.608387 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.608359    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cnvzd\""
Jul 01 14:40:12.617935 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.617909    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"]
Jul 01 14:40:12.634767 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.634736    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-4gdz2"]
Jul 01 14:40:12.636872 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.636846    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.641566 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.641543    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 01 14:40:12.641910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.641887    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 01 14:40:12.642086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.641964    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rq2mq\""
Jul 01 14:40:12.642247 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.642232    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 01 14:40:12.651600 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.651576    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-4gdz2"]
Jul 01 14:40:12.668761 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.668728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.668903 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.668802    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.668903 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.668843    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5rkv\" (UniqueName: \"kubernetes.io/projected/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-api-access-q5rkv\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.669015 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.668917    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.669015 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.668998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8wbl\" (UniqueName: \"kubernetes.io/projected/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-kube-api-access-r8wbl\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.669098 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.669048    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.669181 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.669129    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.669249 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.669193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf0c1ee1-a21a-4358-832d-b627e5a823fe-metrics-client-ca\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.669249 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.669224    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.669326 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.669247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf0c1ee1-a21a-4358-832d-b627e5a823fe-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.677502 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.677472    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-28696"]
Jul 01 14:40:12.679919 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.679902    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.682243 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.682220    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 14:40:12.682363 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.682343    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 14:40:12.682579 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.682559    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 14:40:12.682713 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.682625    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wh7mc\""
Jul 01 14:40:12.770203 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770168    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-accelerators-collector-config\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.770412 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.770412 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770250    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.770412 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.770412 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770397    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e3b7688-df75-43c7-91f1-be8eea205143-root\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.770597 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770424    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-tls\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.770597 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:40:12.770481    2558 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 01 14:40:12.770597 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770476    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf0c1ee1-a21a-4358-832d-b627e5a823fe-metrics-client-ca\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.770597 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770512    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.770597 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:40:12.770546    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-tls podName:bf0c1ee1-a21a-4358-832d-b627e5a823fe nodeName:}" failed. No retries permitted until 2026-07-01 14:40:13.270527838 +0000 UTC m=+61.331107008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-4gdz2" (UID: "bf0c1ee1-a21a-4358-832d-b627e5a823fe") : secret "kube-state-metrics-tls" not found
Jul 01 14:40:12.770597 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf0c1ee1-a21a-4358-832d-b627e5a823fe-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.770597 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e3b7688-df75-43c7-91f1-be8eea205143-metrics-client-ca\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.770982 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770622    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2rpz\" (UniqueName: \"kubernetes.io/projected/6e3b7688-df75-43c7-91f1-be8eea205143-kube-api-access-q2rpz\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.770982 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-wtmp\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.770982 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770682    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.770982 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.770982 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5rkv\" (UniqueName: \"kubernetes.io/projected/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-api-access-q5rkv\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.770982 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770966    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e3b7688-df75-43c7-91f1-be8eea205143-sys\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.771268 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.770996    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.771268 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.771040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8wbl\" (UniqueName: \"kubernetes.io/projected/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-kube-api-access-r8wbl\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.771268 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.771076    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf0c1ee1-a21a-4358-832d-b627e5a823fe-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.771268 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.771080    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-textfile\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.771268 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.771222    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.771268 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.771247    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf0c1ee1-a21a-4358-832d-b627e5a823fe-metrics-client-ca\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.771950 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.771915    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.774091 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.774060    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.774187 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.774151    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.774187 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.774167    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.783402 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.783379    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8wbl\" (UniqueName: \"kubernetes.io/projected/04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4-kube-api-access-r8wbl\") pod \"openshift-state-metrics-57f99c8b49-mrgzr\" (UID: \"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.783550 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.783530    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5rkv\" (UniqueName: \"kubernetes.io/projected/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-api-access-q5rkv\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:12.872159 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872073    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e3b7688-df75-43c7-91f1-be8eea205143-metrics-client-ca\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872159 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872109    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2rpz\" (UniqueName: \"kubernetes.io/projected/6e3b7688-df75-43c7-91f1-be8eea205143-kube-api-access-q2rpz\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872159 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-wtmp\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872172    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e3b7688-df75-43c7-91f1-be8eea205143-sys\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-textfile\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872260    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-accelerators-collector-config\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872291    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872300    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-wtmp\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872301    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e3b7688-df75-43c7-91f1-be8eea205143-sys\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e3b7688-df75-43c7-91f1-be8eea205143-root\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872403    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-tls\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872846 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872563    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-textfile\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872846 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872620    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e3b7688-df75-43c7-91f1-be8eea205143-root\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872846 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872779    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e3b7688-df75-43c7-91f1-be8eea205143-metrics-client-ca\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.872939 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.872902    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-accelerators-collector-config\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.875034 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.875009    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-tls\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.875156 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.875136    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e3b7688-df75-43c7-91f1-be8eea205143-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.882123 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.882100    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2rpz\" (UniqueName: \"kubernetes.io/projected/6e3b7688-df75-43c7-91f1-be8eea205143-kube-api-access-q2rpz\") pod \"node-exporter-28696\" (UID: \"6e3b7688-df75-43c7-91f1-be8eea205143\") " pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:12.915077 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.915038    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"
Jul 01 14:40:12.990069 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:12.990033    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-28696"
Jul 01 14:40:13.276377 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.276295    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:13.279393 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.279362    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0c1ee1-a21a-4358-832d-b627e5a823fe-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-4gdz2\" (UID: \"bf0c1ee1-a21a-4358-832d-b627e5a823fe\") " pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:13.504413 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.504385    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtdlp"
Jul 01 14:40:13.547068 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.546985    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2"
Jul 01 14:40:13.601790 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.601658    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28696" event={"ID":"6e3b7688-df75-43c7-91f1-be8eea205143","Type":"ContainerStarted","Data":"b6fbddbe14b0996bbb9a0dee20a215b4bb81b796f93fd2b1a26fb127f38a6fe1"}
Jul 01 14:40:13.682288 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.682088    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:13.682288 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.682159    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:13.683255 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.683232    2558 patch_prober.go:28] interesting pod/console-9f9dbd8fb-9lmcf container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body=
Jul 01 14:40:13.683352 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.683282    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-9f9dbd8fb-9lmcf" podUID="97e72214-c19b-401e-a6ed-f807439a575a" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused"
Jul 01 14:40:13.699034 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.699001    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:40:13.701511 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.701490    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.704265 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.704067    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 14:40:13.704265 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.704104    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 14:40:13.704265 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.704209    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 14:40:13.704548 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.704503    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 14:40:13.704620 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.704602    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 14:40:13.705138 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.704900    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 14:40:13.705138 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.704965    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 14:40:13.705138 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.704975    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xzr9c\""
Jul 01 14:40:13.705138 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.705079    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 14:40:13.705405 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.705307    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 14:40:13.720383 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.720359    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr"]
Jul 01 14:40:13.725236 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.725208    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:40:13.726433 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:13.726400    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cd3d1a_8124_458a_8a0a_cd6eb48e4ae4.slice/crio-c856e173b4a71769bab8f21e04930e9579aa4d0c4947d160aab891cacef3f180 WatchSource:0}: Error finding container c856e173b4a71769bab8f21e04930e9579aa4d0c4947d160aab891cacef3f180: Status 404 returned error can't find the container with id c856e173b4a71769bab8f21e04930e9579aa4d0c4947d160aab891cacef3f180
Jul 01 14:40:13.740051 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.740031    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-4gdz2"]
Jul 01 14:40:13.743355 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:13.743328    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0c1ee1_a21a_4358_832d_b627e5a823fe.slice/crio-659c751be88c512b3e6317033907422921adb5aa979b57d34283b00982b07879 WatchSource:0}: Error finding container 659c751be88c512b3e6317033907422921adb5aa979b57d34283b00982b07879: Status 404 returned error can't find the container with id 659c751be88c512b3e6317033907422921adb5aa979b57d34283b00982b07879
Jul 01 14:40:13.781048 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781022    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/177046a9-1336-40b8-9861-bf9ed063ed47-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781168 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177046a9-1336-40b8-9861-bf9ed063ed47-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781342 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-config-volume\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781420 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781358    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781420 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781376    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781420 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781400    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46th\" (UniqueName: \"kubernetes.io/projected/177046a9-1336-40b8-9861-bf9ed063ed47-kube-api-access-f46th\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781570 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781443    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781570 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781489    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-web-config\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781570 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781570 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/177046a9-1336-40b8-9861-bf9ed063ed47-config-out\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781790 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781588    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/177046a9-1336-40b8-9861-bf9ed063ed47-tls-assets\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781790 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781607    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177046a9-1336-40b8-9861-bf9ed063ed47-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.781790 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.781627    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.882695 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.882652    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-config-volume\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.882855 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.882713    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.882855 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.882734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.882855 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.882753    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46th\" (UniqueName: \"kubernetes.io/projected/177046a9-1336-40b8-9861-bf9ed063ed47-kube-api-access-f46th\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.882855 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.882781    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.882855 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.882806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-web-config\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.882855 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.882846    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.883159 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.882866    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/177046a9-1336-40b8-9861-bf9ed063ed47-config-out\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.883159 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.882884    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/177046a9-1336-40b8-9861-bf9ed063ed47-tls-assets\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.883159 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:40:13.882885    2558 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 01 14:40:13.883159 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:40:13.882957    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-main-tls podName:177046a9-1336-40b8-9861-bf9ed063ed47 nodeName:}" failed. No retries permitted until 2026-07-01 14:40:14.38293664 +0000 UTC m=+62.443515810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "177046a9-1336-40b8-9861-bf9ed063ed47") : secret "alertmanager-main-tls" not found
Jul 01 14:40:13.884098 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.883427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177046a9-1336-40b8-9861-bf9ed063ed47-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.884098 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.883479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.884098 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.883528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/177046a9-1336-40b8-9861-bf9ed063ed47-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.884098 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.883554    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177046a9-1336-40b8-9861-bf9ed063ed47-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.884098 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:40:13.883737    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/177046a9-1336-40b8-9861-bf9ed063ed47-alertmanager-trusted-ca-bundle podName:177046a9-1336-40b8-9861-bf9ed063ed47 nodeName:}" failed. No retries permitted until 2026-07-01 14:40:14.383720199 +0000 UTC m=+62.444299385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/177046a9-1336-40b8-9861-bf9ed063ed47-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "177046a9-1336-40b8-9861-bf9ed063ed47") : configmap references non-existent config key: ca-bundle.crt
Jul 01 14:40:13.884574 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.884287    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177046a9-1336-40b8-9861-bf9ed063ed47-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.884843 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.884775    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/177046a9-1336-40b8-9861-bf9ed063ed47-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.885927 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.885902    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.886071 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.886035    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.887345 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.887122    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/177046a9-1336-40b8-9861-bf9ed063ed47-config-out\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.887438 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.887367    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-config-volume\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.887590 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.887568    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/177046a9-1336-40b8-9861-bf9ed063ed47-tls-assets\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.887879 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.887859    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.887965 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.887880    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.888080 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.888044    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-web-config\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:13.894159 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:13.894136    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46th\" (UniqueName: \"kubernetes.io/projected/177046a9-1336-40b8-9861-bf9ed063ed47-kube-api-access-f46th\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:14.388695 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.388646    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177046a9-1336-40b8-9861-bf9ed063ed47-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:14.388916 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.388837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:14.389632 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.389599    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177046a9-1336-40b8-9861-bf9ed063ed47-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:14.391557 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.391536    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/177046a9-1336-40b8-9861-bf9ed063ed47-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"177046a9-1336-40b8-9861-bf9ed063ed47\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:14.605748 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.605716    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2" event={"ID":"bf0c1ee1-a21a-4358-832d-b627e5a823fe","Type":"ContainerStarted","Data":"659c751be88c512b3e6317033907422921adb5aa979b57d34283b00982b07879"}
Jul 01 14:40:14.607610 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.607583    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vwk6n" event={"ID":"13243029-0c8d-490c-a21b-437e7ca1b85d","Type":"ContainerStarted","Data":"13ad57772fc98228f94d8c267668447aac781b2f9e67404f8da9e603c422edbe"}
Jul 01 14:40:14.609221 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.609199    2558 generic.go:356] "Generic (PLEG): container finished" podID="6e3b7688-df75-43c7-91f1-be8eea205143" containerID="1e26e6ae3833d76d044ef9011c23743eaaafa5750902511088baa80a50388c23" exitCode=0
Jul 01 14:40:14.609332 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.609268    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28696" event={"ID":"6e3b7688-df75-43c7-91f1-be8eea205143","Type":"ContainerDied","Data":"1e26e6ae3833d76d044ef9011c23743eaaafa5750902511088baa80a50388c23"}
Jul 01 14:40:14.614448 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.611298    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr" event={"ID":"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4","Type":"ContainerStarted","Data":"14c29e6cd1a2062ae39a4d903d5613c836e612eb6ade0fafa37d494fe26d2692"}
Jul 01 14:40:14.614448 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.611426    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr" event={"ID":"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4","Type":"ContainerStarted","Data":"bfbe0384f0773241f5cdade51459664fdaae0b4514ad93618da88e043ec5f4cf"}
Jul 01 14:40:14.614448 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.611475    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr" event={"ID":"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4","Type":"ContainerStarted","Data":"c856e173b4a71769bab8f21e04930e9579aa4d0c4947d160aab891cacef3f180"}
Jul 01 14:40:14.615026 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.615006    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:14.652064 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.652003    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vwk6n" podStartSLOduration=2.179010236 podStartE2EDuration="12.651981697s" totalImagesPullingTime="10.472971461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:40:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:14.651489779 +0000 UTC m=+62.712068972" watchObservedRunningTime="2026-07-01 14:40:14.651981697 +0000 UTC m=+62.712560889"
Jul 01 14:40:14.759431 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:14.759397    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:40:14.762863 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:14.762840    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177046a9_1336_40b8_9861_bf9ed063ed47.slice/crio-0a80973ae8ecf5666c57a618b1050331f053730f6ede40c2172b3cb7ae33713f WatchSource:0}: Error finding container 0a80973ae8ecf5666c57a618b1050331f053730f6ede40c2172b3cb7ae33713f: Status 404 returned error can't find the container with id 0a80973ae8ecf5666c57a618b1050331f053730f6ede40c2172b3cb7ae33713f
Jul 01 14:40:15.615962 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.615923    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"177046a9-1336-40b8-9861-bf9ed063ed47","Type":"ContainerStarted","Data":"0a80973ae8ecf5666c57a618b1050331f053730f6ede40c2172b3cb7ae33713f"}
Jul 01 14:40:15.618060 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.618036    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28696" event={"ID":"6e3b7688-df75-43c7-91f1-be8eea205143","Type":"ContainerStarted","Data":"cde22e1eae6075e6ef5ef14143773ab0aedbd7d2a43dc143777fbcbfaf7921b4"}
Jul 01 14:40:15.618183 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.618081    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28696" event={"ID":"6e3b7688-df75-43c7-91f1-be8eea205143","Type":"ContainerStarted","Data":"9abb33c0ffcea3feaf174d410958b9851eac342deb54c5f84e42c7bbac39a2bb"}
Jul 01 14:40:15.620131 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.620107    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr" event={"ID":"04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4","Type":"ContainerStarted","Data":"5326f6cb440dcebf7daf65881bb5fb5a186274c19e24f0a499520428df102b6d"}
Jul 01 14:40:15.639643 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.639606    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-28696" podStartSLOduration=2.710529445 podStartE2EDuration="3.639592107s" totalImagesPullingTime="862.872597ms" totalInitContainerRuntime="66.190065ms" isStatefulPod=true podCreationTimestamp="2026-07-01 14:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:15.63813595 +0000 UTC m=+63.698715142" watchObservedRunningTime="2026-07-01 14:40:15.639592107 +0000 UTC m=+63.700171334"
Jul 01 14:40:15.660786 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.660732    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-mrgzr" podStartSLOduration=2.060179845 podStartE2EDuration="3.660713541s" totalImagesPullingTime="1.600533696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:15.659395361 +0000 UTC m=+63.719974554" watchObservedRunningTime="2026-07-01 14:40:15.660713541 +0000 UTC m=+63.721292734"
Jul 01 14:40:15.723032 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.722972    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"]
Jul 01 14:40:15.725370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.725354    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.727655 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.727634    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 01 14:40:15.727764 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.727722    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 01 14:40:15.727838 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.727728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8tp2x\""
Jul 01 14:40:15.727891 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.727869    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 01 14:40:15.728008 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.727989    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 01 14:40:15.728203 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.728189    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1a6fc6ugk62bd\""
Jul 01 14:40:15.728339 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.728324    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 01 14:40:15.738342 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.738324    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"]
Jul 01 14:40:15.803876 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.803851    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krqd\" (UniqueName: \"kubernetes.io/projected/513d4970-c815-4490-a77d-c5b35ef74bc3-kube-api-access-4krqd\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.804162 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.803882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-tls\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.804162 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.803900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.804162 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.803931    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513d4970-c815-4490-a77d-c5b35ef74bc3-metrics-client-ca\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.804162 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.804024    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.804162 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.804056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-grpc-tls\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.804162 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.804087    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.804162 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.804111    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.905134 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.905103    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.905134 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.905135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-grpc-tls\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.905283 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.905159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.905283 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.905182    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.905356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.905320    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krqd\" (UniqueName: \"kubernetes.io/projected/513d4970-c815-4490-a77d-c5b35ef74bc3-kube-api-access-4krqd\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.905399 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.905365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-tls\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.905453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.905396    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.905500 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.905453    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513d4970-c815-4490-a77d-c5b35ef74bc3-metrics-client-ca\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.906238 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.906216    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513d4970-c815-4490-a77d-c5b35ef74bc3-metrics-client-ca\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.908351 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.908302    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.908351 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.908334    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-tls\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.908480 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.908390    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.908480 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.908411    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.908545 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.908477    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.908788 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.908769    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/513d4970-c815-4490-a77d-c5b35ef74bc3-secret-grpc-tls\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:15.918619 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:15.918600    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krqd\" (UniqueName: \"kubernetes.io/projected/513d4970-c815-4490-a77d-c5b35ef74bc3-kube-api-access-4krqd\") pod \"thanos-querier-5dd8d4bdfd-4mn75\" (UID: \"513d4970-c815-4490-a77d-c5b35ef74bc3\") " pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:16.035013 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:16.034954    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:16.164300 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:16.164272    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"]
Jul 01 14:40:16.168575 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:16.168549    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513d4970_c815_4490_a77d_c5b35ef74bc3.slice/crio-fa65afdc63aa38c17ad69f6c010bf1e55f3b6f55730c8a9f70d538beff61ea1d WatchSource:0}: Error finding container fa65afdc63aa38c17ad69f6c010bf1e55f3b6f55730c8a9f70d538beff61ea1d: Status 404 returned error can't find the container with id fa65afdc63aa38c17ad69f6c010bf1e55f3b6f55730c8a9f70d538beff61ea1d
Jul 01 14:40:16.623844 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:16.623797    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75" event={"ID":"513d4970-c815-4490-a77d-c5b35ef74bc3","Type":"ContainerStarted","Data":"fa65afdc63aa38c17ad69f6c010bf1e55f3b6f55730c8a9f70d538beff61ea1d"}
Jul 01 14:40:17.074658 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.074627    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-85b4d78d6f-768rp"]
Jul 01 14:40:17.077992 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.077976    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.080457 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.080436    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 14:40:17.080591 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.080436    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 01 14:40:17.081443 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.081402    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 01 14:40:17.081443 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.081422    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kjdhv\""
Jul 01 14:40:17.081443 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.081426    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a0071o5s1cqgr\""
Jul 01 14:40:17.081642 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.081426    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 01 14:40:17.095750 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.095731    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85b4d78d6f-768rp"]
Jul 01 14:40:17.215672 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.215638    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d892e1ea-9805-4c7a-bf8f-525cc3b52782-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.215848 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.215730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d892e1ea-9805-4c7a-bf8f-525cc3b52782-metrics-server-audit-profiles\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.215848 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.215763    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwqr\" (UniqueName: \"kubernetes.io/projected/d892e1ea-9805-4c7a-bf8f-525cc3b52782-kube-api-access-vnwqr\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.215848 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.215805    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d892e1ea-9805-4c7a-bf8f-525cc3b52782-secret-metrics-server-client-certs\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.216026 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.215878    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d892e1ea-9805-4c7a-bf8f-525cc3b52782-audit-log\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.216026 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.215961    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:40:17.216026 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.216008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:40:17.216163 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.216055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d892e1ea-9805-4c7a-bf8f-525cc3b52782-secret-metrics-server-tls\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.216163 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.216079    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d892e1ea-9805-4c7a-bf8f-525cc3b52782-client-ca-bundle\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.218760 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.218732    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 14:40:17.218760 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.218760    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 14:40:17.229659 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.229637    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388279eb-7ee2-4130-9f63-3f54e10c7419-metrics-certs\") pod \"network-metrics-daemon-khdj5\" (UID: \"388279eb-7ee2-4130-9f63-3f54e10c7419\") " pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:40:17.229781 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.229642    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cd6a69f-c352-40c4-a8b9-125145b3ec48-original-pull-secret\") pod \"global-pull-secret-syncer-t85xw\" (UID: \"3cd6a69f-c352-40c4-a8b9-125145b3ec48\") " pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:40:17.317246 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.317212    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d892e1ea-9805-4c7a-bf8f-525cc3b52782-audit-log\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.317396 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.317273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d892e1ea-9805-4c7a-bf8f-525cc3b52782-secret-metrics-server-tls\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.317396 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.317290    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d892e1ea-9805-4c7a-bf8f-525cc3b52782-client-ca-bundle\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.317396 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.317346    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d892e1ea-9805-4c7a-bf8f-525cc3b52782-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.317396 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.317377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bnk\" (UniqueName: \"kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk\") pod \"network-check-target-llw6c\" (UID: \"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c\") " pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:40:17.317653 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.317409    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d892e1ea-9805-4c7a-bf8f-525cc3b52782-metrics-server-audit-profiles\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.317653 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.317434    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwqr\" (UniqueName: \"kubernetes.io/projected/d892e1ea-9805-4c7a-bf8f-525cc3b52782-kube-api-access-vnwqr\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.317653 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.317471    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d892e1ea-9805-4c7a-bf8f-525cc3b52782-secret-metrics-server-client-certs\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.318208 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.317672    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d892e1ea-9805-4c7a-bf8f-525cc3b52782-audit-log\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.318449 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.318403    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d892e1ea-9805-4c7a-bf8f-525cc3b52782-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.318663 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.318628    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d892e1ea-9805-4c7a-bf8f-525cc3b52782-metrics-server-audit-profiles\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.319928 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.319908    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 14:40:17.320023 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.319968    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d892e1ea-9805-4c7a-bf8f-525cc3b52782-secret-metrics-server-tls\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.320241 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.320212    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d892e1ea-9805-4c7a-bf8f-525cc3b52782-secret-metrics-server-client-certs\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.320316 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.320285    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d892e1ea-9805-4c7a-bf8f-525cc3b52782-client-ca-bundle\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.331135 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.331056    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 14:40:17.332183 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.332151    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwqr\" (UniqueName: \"kubernetes.io/projected/d892e1ea-9805-4c7a-bf8f-525cc3b52782-kube-api-access-vnwqr\") pod \"metrics-server-85b4d78d6f-768rp\" (UID: \"d892e1ea-9805-4c7a-bf8f-525cc3b52782\") " pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.340106 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.340058    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8bnk\" (UniqueName: \"kubernetes.io/projected/c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c-kube-api-access-l8bnk\") pod \"network-check-target-llw6c\" (UID: \"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c\") " pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:40:17.387740 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.387709    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:17.397834 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.397805    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"]
Jul 01 14:40:17.408621 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.408599    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9f9dbd8fb-9lmcf"]
Jul 01 14:40:17.408767 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.408744    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"
Jul 01 14:40:17.411057 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.410873    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 01 14:40:17.411195 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.411171    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qgfsz\""
Jul 01 14:40:17.411717 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.411424    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"]
Jul 01 14:40:17.437817 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.437790    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ff88cbc7d-pr6jr"]
Jul 01 14:40:17.460105 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.460083    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5778q\""
Jul 01 14:40:17.463999 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.463980    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t85xw"
Jul 01 14:40:17.468158 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.468086    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:40:17.470258 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.470236    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hvdsl\""
Jul 01 14:40:17.470852 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.470830    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff88cbc7d-pr6jr"]
Jul 01 14:40:17.470989 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.470966    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.478215 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.478175    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khdj5"
Jul 01 14:40:17.519317 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.519208    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-trusted-ca-bundle\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.519317 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.519271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-oauth-serving-cert\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.519485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.519329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-console-config\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.519485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.519375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-oauth-config\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.519485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.519406    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-serving-cert\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.519485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.519438    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-service-ca\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.519699 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.519483    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlks\" (UniqueName: \"kubernetes.io/projected/13006537-9f80-448e-aa88-de938003c5de-kube-api-access-8hlks\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.519699 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.519539    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c52135b-b9ab-416a-ba90-3f2082481551-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-6kb5c\" (UID: \"4c52135b-b9ab-416a-ba90-3f2082481551\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"
Jul 01 14:40:17.519699 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.519564    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q656s\" (UniqueName: \"kubernetes.io/projected/4c52135b-b9ab-416a-ba90-3f2082481551-kube-api-access-q656s\") pod \"monitoring-plugin-cd8656b8-6kb5c\" (UID: \"4c52135b-b9ab-416a-ba90-3f2082481551\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"
Jul 01 14:40:17.620285 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.620192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-trusted-ca-bundle\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.620285 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.620241    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-oauth-serving-cert\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.620285 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.620281    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-console-config\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.620540 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.620311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-oauth-config\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.620540 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.620342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-serving-cert\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.620540 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.620374    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-service-ca\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.620540 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.620399    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlks\" (UniqueName: \"kubernetes.io/projected/13006537-9f80-448e-aa88-de938003c5de-kube-api-access-8hlks\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.620540 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.620432    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c52135b-b9ab-416a-ba90-3f2082481551-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-6kb5c\" (UID: \"4c52135b-b9ab-416a-ba90-3f2082481551\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"
Jul 01 14:40:17.620540 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.620458    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q656s\" (UniqueName: \"kubernetes.io/projected/4c52135b-b9ab-416a-ba90-3f2082481551-kube-api-access-q656s\") pod \"monitoring-plugin-cd8656b8-6kb5c\" (UID: \"4c52135b-b9ab-416a-ba90-3f2082481551\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"
Jul 01 14:40:17.621197 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.621137    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-console-config\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.621334 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.621276    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-trusted-ca-bundle\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.621334 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.621292    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-oauth-serving-cert\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.621945 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.621907    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-service-ca\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.623597 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.623574    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-oauth-config\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.623711 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.623646    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-serving-cert\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.623771 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.623738    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c52135b-b9ab-416a-ba90-3f2082481551-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-6kb5c\" (UID: \"4c52135b-b9ab-416a-ba90-3f2082481551\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"
Jul 01 14:40:17.631343 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.631301    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlks\" (UniqueName: \"kubernetes.io/projected/13006537-9f80-448e-aa88-de938003c5de-kube-api-access-8hlks\") pod \"console-6ff88cbc7d-pr6jr\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") " pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:17.631494 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.631472    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q656s\" (UniqueName: \"kubernetes.io/projected/4c52135b-b9ab-416a-ba90-3f2082481551-kube-api-access-q656s\") pod \"monitoring-plugin-cd8656b8-6kb5c\" (UID: \"4c52135b-b9ab-416a-ba90-3f2082481551\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"
Jul 01 14:40:17.721744 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.721711    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"
Jul 01 14:40:17.784622 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:17.784582    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:18.015110 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:18.014940    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-llw6c"]
Jul 01 14:40:18.091532 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:18.091481    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff88cbc7d-pr6jr"]
Jul 01 14:40:18.193102 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:18.193027    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c4c5b8_1ba3_41f3_bee3_cf74ae04595c.slice/crio-3c373c1b59d0dfbfd582406f0298f7138c303d4c4388006cc0ae79b2cc7e5d0e WatchSource:0}: Error finding container 3c373c1b59d0dfbfd582406f0298f7138c303d4c4388006cc0ae79b2cc7e5d0e: Status 404 returned error can't find the container with id 3c373c1b59d0dfbfd582406f0298f7138c303d4c4388006cc0ae79b2cc7e5d0e
Jul 01 14:40:18.193876 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:18.193848    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13006537_9f80_448e_aa88_de938003c5de.slice/crio-170666771a983f146267a60c11058ecb21b906f81e3d56ec5cf5dbe60b04d84a WatchSource:0}: Error finding container 170666771a983f146267a60c11058ecb21b906f81e3d56ec5cf5dbe60b04d84a: Status 404 returned error can't find the container with id 170666771a983f146267a60c11058ecb21b906f81e3d56ec5cf5dbe60b04d84a
Jul 01 14:40:18.230726 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:18.230705    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"]
Jul 01 14:40:18.233714 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:18.233679    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-khdj5"]
Jul 01 14:40:18.267115 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:18.267084    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t85xw"]
Jul 01 14:40:18.272108 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:18.272085    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85b4d78d6f-768rp"]
Jul 01 14:40:18.633484 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:18.633441    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-llw6c" event={"ID":"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c","Type":"ContainerStarted","Data":"3c373c1b59d0dfbfd582406f0298f7138c303d4c4388006cc0ae79b2cc7e5d0e"}
Jul 01 14:40:18.634754 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:18.634722    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff88cbc7d-pr6jr" event={"ID":"13006537-9f80-448e-aa88-de938003c5de","Type":"ContainerStarted","Data":"170666771a983f146267a60c11058ecb21b906f81e3d56ec5cf5dbe60b04d84a"}
Jul 01 14:40:18.829334 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:18.829280    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c52135b_b9ab_416a_ba90_3f2082481551.slice/crio-9227d07463accb390a06cce7f161e1b1e2382e77dbbbc1d4e4e9f06e46d087ba WatchSource:0}: Error finding container 9227d07463accb390a06cce7f161e1b1e2382e77dbbbc1d4e4e9f06e46d087ba: Status 404 returned error can't find the container with id 9227d07463accb390a06cce7f161e1b1e2382e77dbbbc1d4e4e9f06e46d087ba
Jul 01 14:40:18.830044 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:18.829998    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388279eb_7ee2_4130_9f63_3f54e10c7419.slice/crio-42aa16a8bd771efca67bf6384376bfbcfff618e9fc901eed421b4c0cc32ae36a WatchSource:0}: Error finding container 42aa16a8bd771efca67bf6384376bfbcfff618e9fc901eed421b4c0cc32ae36a: Status 404 returned error can't find the container with id 42aa16a8bd771efca67bf6384376bfbcfff618e9fc901eed421b4c0cc32ae36a
Jul 01 14:40:18.831122 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:18.831060    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd6a69f_c352_40c4_a8b9_125145b3ec48.slice/crio-1df4203d48f0e212937ce3e7ec48eb8fe0ecc4df8bf854f95238e1a5b013ac1f WatchSource:0}: Error finding container 1df4203d48f0e212937ce3e7ec48eb8fe0ecc4df8bf854f95238e1a5b013ac1f: Status 404 returned error can't find the container with id 1df4203d48f0e212937ce3e7ec48eb8fe0ecc4df8bf854f95238e1a5b013ac1f
Jul 01 14:40:18.832497 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:18.832415    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd892e1ea_9805_4c7a_bf8f_525cc3b52782.slice/crio-04295f537ca6ab0d00a94c62ad41e236c2cf579f9f7a1a230e75c5160a985005 WatchSource:0}: Error finding container 04295f537ca6ab0d00a94c62ad41e236c2cf579f9f7a1a230e75c5160a985005: Status 404 returned error can't find the container with id 04295f537ca6ab0d00a94c62ad41e236c2cf579f9f7a1a230e75c5160a985005
Jul 01 14:40:19.044606 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.044575    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:40:19.062140 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.059519    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.066453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.064584    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 14:40:19.066453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.064896    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 14:40:19.066453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.065083    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 14:40:19.066453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.065484    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 14:40:19.066453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.065669    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7654uonh8jpqh\""
Jul 01 14:40:19.066453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.065901    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 14:40:19.066453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.066147    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 14:40:19.066453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.066322    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 14:40:19.067100 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.066516    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 14:40:19.067100 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.066728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 14:40:19.067100 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.066917    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 14:40:19.069065 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.069042    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:40:19.070455 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.069526    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 14:40:19.070455 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.069747    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 14:40:19.070455 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.070313    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d26r5\""
Jul 01 14:40:19.070455 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.070411    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 14:40:19.074320 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.071867    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 14:40:19.134333 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134366    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134424    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134452    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134478    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134504    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0342580-1dea-4f27-8a3e-0e940b19bd5d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134583    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zlft\" (UniqueName: \"kubernetes.io/projected/a0342580-1dea-4f27-8a3e-0e940b19bd5d-kube-api-access-4zlft\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134608    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134643    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134723    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.134772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134755    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.135350 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134784    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-config\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.135350 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134808    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0342580-1dea-4f27-8a3e-0e940b19bd5d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.135350 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134858    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.135350 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.135350 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.135350 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.134969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.135350 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.135033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.235986    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236029    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236055    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236080    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236145    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236212    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236247    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236266    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236290    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236319    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0342580-1dea-4f27-8a3e-0e940b19bd5d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236344    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236368    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zlft\" (UniqueName: \"kubernetes.io/projected/a0342580-1dea-4f27-8a3e-0e940b19bd5d-kube-api-access-4zlft\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236457    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.236795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.237710 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236511    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.237710 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-config\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.237710 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236545    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.237710 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.236571    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0342580-1dea-4f27-8a3e-0e940b19bd5d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.239073 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.238280    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.239073 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.238762    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.240118 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.240076    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0342580-1dea-4f27-8a3e-0e940b19bd5d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.240209 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.240185    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.240354 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.240329    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.240901 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.240874    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.241590 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.241559    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.242381 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.241946    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.242381 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.241974    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.242381 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.242347    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-config\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.243158 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.242988    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.243158 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.243120    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.244165 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.243804    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.246058 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.244940    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0342580-1dea-4f27-8a3e-0e940b19bd5d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.246475 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.246379    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0342580-1dea-4f27-8a3e-0e940b19bd5d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.246590 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.246566    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.246708 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.246575    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.247227 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.247165    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0342580-1dea-4f27-8a3e-0e940b19bd5d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.252592 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.252570    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zlft\" (UniqueName: \"kubernetes.io/projected/a0342580-1dea-4f27-8a3e-0e940b19bd5d-kube-api-access-4zlft\") pod \"prometheus-k8s-0\" (UID: \"a0342580-1dea-4f27-8a3e-0e940b19bd5d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.382931 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.382892    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:19.647416 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.643979    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2" event={"ID":"bf0c1ee1-a21a-4358-832d-b627e5a823fe","Type":"ContainerStarted","Data":"03ed2d2a64512f609027a98eca228988a676b9f1b613691f4df1e63dc95b8906"}
Jul 01 14:40:19.649290 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.647737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2" event={"ID":"bf0c1ee1-a21a-4358-832d-b627e5a823fe","Type":"ContainerStarted","Data":"8d909d4ca2d3e5fe61fd808dc38c4d1a5b85ff86edccda54583dc96647bdc86c"}
Jul 01 14:40:19.649290 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.647807    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2" event={"ID":"bf0c1ee1-a21a-4358-832d-b627e5a823fe","Type":"ContainerStarted","Data":"dc535ebb1f272ff2294dc3460ad432b21ef632da60cb2c95f9165d1df46c29c8"}
Jul 01 14:40:19.653372 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.653328    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c" event={"ID":"4c52135b-b9ab-416a-ba90-3f2082481551","Type":"ContainerStarted","Data":"9227d07463accb390a06cce7f161e1b1e2382e77dbbbc1d4e4e9f06e46d087ba"}
Jul 01 14:40:19.655552 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.655454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t85xw" event={"ID":"3cd6a69f-c352-40c4-a8b9-125145b3ec48","Type":"ContainerStarted","Data":"1df4203d48f0e212937ce3e7ec48eb8fe0ecc4df8bf854f95238e1a5b013ac1f"}
Jul 01 14:40:19.660730 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.660054    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff88cbc7d-pr6jr" event={"ID":"13006537-9f80-448e-aa88-de938003c5de","Type":"ContainerStarted","Data":"9f9273fea282ba420cbd4d9b3793f61fdf4cd4ec0925fc0a37e516af10d41d75"}
Jul 01 14:40:19.686718 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.683739    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-4gdz2" podStartSLOduration=2.59721497 podStartE2EDuration="7.68371724s" totalImagesPullingTime="5.08650227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:19.676140118 +0000 UTC m=+67.736719313" watchObservedRunningTime="2026-07-01 14:40:19.68371724 +0000 UTC m=+67.744296429"
Jul 01 14:40:19.686718 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.685995    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-khdj5" event={"ID":"388279eb-7ee2-4130-9f63-3f54e10c7419","Type":"ContainerStarted","Data":"42aa16a8bd771efca67bf6384376bfbcfff618e9fc901eed421b4c0cc32ae36a"}
Jul 01 14:40:19.694763 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.694681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp" event={"ID":"d892e1ea-9805-4c7a-bf8f-525cc3b52782","Type":"ContainerStarted","Data":"04295f537ca6ab0d00a94c62ad41e236c2cf579f9f7a1a230e75c5160a985005"}
Jul 01 14:40:19.709912 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.709859    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6ff88cbc7d-pr6jr" podStartSLOduration=2.709835034 podStartE2EDuration="2.709835034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:19.7061542 +0000 UTC m=+67.766733391" watchObservedRunningTime="2026-07-01 14:40:19.709835034 +0000 UTC m=+67.770414227"
Jul 01 14:40:19.745347 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:19.745073    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:40:19.754127 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:40:19.754096    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0342580_1dea_4f27_8a3e_0e940b19bd5d.slice/crio-eadc83ab43ef22a62fc51a85a59f374536702ca97b997603158138e88957d999 WatchSource:0}: Error finding container eadc83ab43ef22a62fc51a85a59f374536702ca97b997603158138e88957d999: Status 404 returned error can't find the container with id eadc83ab43ef22a62fc51a85a59f374536702ca97b997603158138e88957d999
Jul 01 14:40:20.590553 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:20.590513    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l8l96"
Jul 01 14:40:20.704095 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:20.703468    2558 generic.go:356] "Generic (PLEG): container finished" podID="a0342580-1dea-4f27-8a3e-0e940b19bd5d" containerID="bcc0478abdf364906914a999b6c31c7069c0ceeccba186cb39d33d680bd96ce7" exitCode=0
Jul 01 14:40:20.704095 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:20.703602    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0342580-1dea-4f27-8a3e-0e940b19bd5d","Type":"ContainerDied","Data":"bcc0478abdf364906914a999b6c31c7069c0ceeccba186cb39d33d680bd96ce7"}
Jul 01 14:40:20.704095 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:20.703700    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0342580-1dea-4f27-8a3e-0e940b19bd5d","Type":"ContainerStarted","Data":"eadc83ab43ef22a62fc51a85a59f374536702ca97b997603158138e88957d999"}
Jul 01 14:40:20.706409 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:20.706372    2558 generic.go:356] "Generic (PLEG): container finished" podID="177046a9-1336-40b8-9861-bf9ed063ed47" containerID="294d2e64d12f699d88e5ca966026020f40b35a1c230dffb6fe80146f74207b1b" exitCode=0
Jul 01 14:40:20.706511 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:20.706472    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"177046a9-1336-40b8-9861-bf9ed063ed47","Type":"ContainerDied","Data":"294d2e64d12f699d88e5ca966026020f40b35a1c230dffb6fe80146f74207b1b"}
Jul 01 14:40:20.711149 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:20.710954    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75" event={"ID":"513d4970-c815-4490-a77d-c5b35ef74bc3","Type":"ContainerStarted","Data":"1274af41343c20d444b6a8b9483a0ce0fd55fb88522be99d054d2877b1b03795"}
Jul 01 14:40:20.711149 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:20.711019    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75" event={"ID":"513d4970-c815-4490-a77d-c5b35ef74bc3","Type":"ContainerStarted","Data":"87ae266fef77b5eb04815b7260cc86c7ec7d9f9942ecc3e07abbf190d491a355"}
Jul 01 14:40:20.711149 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:20.711059    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75" event={"ID":"513d4970-c815-4490-a77d-c5b35ef74bc3","Type":"ContainerStarted","Data":"c90f19941ce7dac8d7c93606c35b98aea503d9aa37b97e89739ad0055b60f702"}
Jul 01 14:40:22.722670 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:22.722626    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-khdj5" event={"ID":"388279eb-7ee2-4130-9f63-3f54e10c7419","Type":"ContainerStarted","Data":"3989e420b673c3204aec02c054f367561c3cf8d6213a16ab29030222d8bce49a"}
Jul 01 14:40:22.724291 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:22.724263    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp" event={"ID":"d892e1ea-9805-4c7a-bf8f-525cc3b52782","Type":"ContainerStarted","Data":"cd4814588b3f1ea979c5547086f7b2b2cf508c29241ac1e6c03425b8479166b3"}
Jul 01 14:40:22.753898 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:22.753852    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp" podStartSLOduration=2.14912658 podStartE2EDuration="5.753834392s" totalImagesPullingTime="3.604707812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:22.750843169 +0000 UTC m=+70.811422362" watchObservedRunningTime="2026-07-01 14:40:22.753834392 +0000 UTC m=+70.814413563"
Jul 01 14:40:23.736054 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.736016    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"177046a9-1336-40b8-9861-bf9ed063ed47","Type":"ContainerStarted","Data":"ecd5e843f7a53a7173eedd4a134c43c15c369abcb04dc3579aac32f6414d04d7"}
Jul 01 14:40:23.736474 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.736111    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"177046a9-1336-40b8-9861-bf9ed063ed47","Type":"ContainerStarted","Data":"7fb1b57d76f42b8c487f09a432a07800757304779377d12d56f5550da2cb1c28"}
Jul 01 14:40:23.736474 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.736147    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"177046a9-1336-40b8-9861-bf9ed063ed47","Type":"ContainerStarted","Data":"745f41b27dd0481c2764f063afb05fa4bb3f20e88ec90cd4d33b74201f239838"}
Jul 01 14:40:23.736474 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.736172    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"177046a9-1336-40b8-9861-bf9ed063ed47","Type":"ContainerStarted","Data":"4ff1d953b41cbce716630e8488a86ff24cb8b2a3910fdb57c0da9a9044728468"}
Jul 01 14:40:23.736474 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.736207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"177046a9-1336-40b8-9861-bf9ed063ed47","Type":"ContainerStarted","Data":"786cdbcb93afb7a3810ac4cb272e14ef20eb72047678a02fa8f9e3dc658720b8"}
Jul 01 14:40:23.736474 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.736231    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"177046a9-1336-40b8-9861-bf9ed063ed47","Type":"ContainerStarted","Data":"bdfa208bff83151df2f782325f4df85233321fc5ff8f0296ee64f4a6aadf9529"}
Jul 01 14:40:23.739475 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.739435    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75" event={"ID":"513d4970-c815-4490-a77d-c5b35ef74bc3","Type":"ContainerStarted","Data":"6369e83738abbb58a1f6bef5569f46fcada4b5e4fc8aba6755493518c7729df6"}
Jul 01 14:40:23.739604 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.739504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75" event={"ID":"513d4970-c815-4490-a77d-c5b35ef74bc3","Type":"ContainerStarted","Data":"5e9b7a0294a959dc9a21ca16186721a7bc79e2edbb4639152544e75f28f0aa83"}
Jul 01 14:40:23.739604 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.739549    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75" event={"ID":"513d4970-c815-4490-a77d-c5b35ef74bc3","Type":"ContainerStarted","Data":"ffab1ecf8bd5ad7b8c33266ba2c4ffe794ce553322209519b727f9bba7672bef"}
Jul 01 14:40:23.739743 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.739612    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:23.741083 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.741058    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-llw6c" event={"ID":"c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c","Type":"ContainerStarted","Data":"ba77e1e2e6ab947d6bf3b9cbf959dce79b5c1987d380a5ee49e1ee818e431fcb"}
Jul 01 14:40:23.741184 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.741110    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:40:23.742897 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.742876    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c" event={"ID":"4c52135b-b9ab-416a-ba90-3f2082481551","Type":"ContainerStarted","Data":"730a8b5c3d1dc491f64e8c485671aa2ffe32d89a4be17cdae454d3f002d9b1dd"}
Jul 01 14:40:23.743032 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.743009    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"
Jul 01 14:40:23.745924 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.745897    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-khdj5" event={"ID":"388279eb-7ee2-4130-9f63-3f54e10c7419","Type":"ContainerStarted","Data":"a56c34405f8b6413096f9b0fc13cc6b2f26432d4b6fb9024ad112a7a43d6ffba"}
Jul 01 14:40:23.748991 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.748971    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c"
Jul 01 14:40:23.774729 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.774665    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.491923126 podStartE2EDuration="10.77464929s" totalImagesPullingTime="7.25202524s" totalInitContainerRuntime="30.700924ms" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:23.772215373 +0000 UTC m=+71.832794578" watchObservedRunningTime="2026-07-01 14:40:23.77464929 +0000 UTC m=+71.835228501"
Jul 01 14:40:23.799272 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.799219    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-llw6c" podStartSLOduration=66.886882962 podStartE2EDuration="1m11.799197892s" totalImagesPullingTime="4.91231493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:23.794615668 +0000 UTC m=+71.855194909" watchObservedRunningTime="2026-07-01 14:40:23.799197892 +0000 UTC m=+71.859777085"
Jul 01 14:40:23.831769 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.831682    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-cd8656b8-6kb5c" podStartSLOduration=3.222487991 podStartE2EDuration="6.831660748s" totalImagesPullingTime="3.609172757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:23.831171001 +0000 UTC m=+71.891750197" watchObservedRunningTime="2026-07-01 14:40:23.831660748 +0000 UTC m=+71.892239940"
Jul 01 14:40:23.868657 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.868541    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75" podStartSLOduration=2.230277842 podStartE2EDuration="8.868520241s" totalImagesPullingTime="6.638242399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:23.867164897 +0000 UTC m=+71.927744118" watchObservedRunningTime="2026-07-01 14:40:23.868520241 +0000 UTC m=+71.929099434"
Jul 01 14:40:23.889795 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:23.889422    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-khdj5" podStartSLOduration=68.285023288 podStartE2EDuration="1m11.889406967s" totalImagesPullingTime="3.604383679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:23.889155203 +0000 UTC m=+71.949734397" watchObservedRunningTime="2026-07-01 14:40:23.889406967 +0000 UTC m=+71.949986158"
Jul 01 14:40:24.760243 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:24.760163    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5dd8d4bdfd-4mn75"
Jul 01 14:40:27.770535 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:27.770505    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0342580-1dea-4f27-8a3e-0e940b19bd5d","Type":"ContainerStarted","Data":"3c44f2af1af6596116af1dd0430d4667aa3c529ce1160317265cc99178d430d4"}
Jul 01 14:40:27.771062 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:27.770623    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0342580-1dea-4f27-8a3e-0e940b19bd5d","Type":"ContainerStarted","Data":"3ee995b472d5e7604d804301ca2c85d8be1eaa078e42e8f21b92718633a335c9"}
Jul 01 14:40:27.771062 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:27.770670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0342580-1dea-4f27-8a3e-0e940b19bd5d","Type":"ContainerStarted","Data":"0d30c6ce54eda19d41de66c9fccff21517f0a3303fb296f6bb04027decf3ba8e"}
Jul 01 14:40:27.771062 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:27.770733    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0342580-1dea-4f27-8a3e-0e940b19bd5d","Type":"ContainerStarted","Data":"a62ab78bd1fdd7a388a6731557b22cfc77a49968e14dd4987a9d931072049b98"}
Jul 01 14:40:27.772466 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:27.772356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t85xw" event={"ID":"3cd6a69f-c352-40c4-a8b9-125145b3ec48","Type":"ContainerStarted","Data":"54aa33e8de61fd3e0b6af6648a4b1909d220df79d087be5d656c65e81d126047"}
Jul 01 14:40:27.785619 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:27.785594    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:27.785750 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:27.785727    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:27.791585 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:27.791561    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:27.799957 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:27.799914    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t85xw" podStartSLOduration=67.223204784 podStartE2EDuration="1m15.799898933s" totalImagesPullingTime="8.576694149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:27.797959873 +0000 UTC m=+75.858539065" watchObservedRunningTime="2026-07-01 14:40:27.799898933 +0000 UTC m=+75.860478124"
Jul 01 14:40:28.782822 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:28.782783    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0342580-1dea-4f27-8a3e-0e940b19bd5d","Type":"ContainerStarted","Data":"344060fba1d1d2ec0df85d98f892558b45f5d38b7cf73f184b3d5163a9216b79"}
Jul 01 14:40:28.783381 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:28.783346    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0342580-1dea-4f27-8a3e-0e940b19bd5d","Type":"ContainerStarted","Data":"2ac837a7df35241582e68c1541df2ce6ebbd4c5c50899c664785278d87082f8b"}
Jul 01 14:40:28.786581 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:28.786562    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:40:28.817649 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:28.817597    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.08145531 podStartE2EDuration="9.817581392s" totalImagesPullingTime="6.703152746s" totalInitContainerRuntime="32.973336ms" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:28.816614975 +0000 UTC m=+76.877194205" watchObservedRunningTime="2026-07-01 14:40:28.817581392 +0000 UTC m=+76.878160583"
Jul 01 14:40:29.384123 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:29.384082    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:37.388294 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:37.388192    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:37.388294 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:37.388251    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:42.432131 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:42.432072    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-9f9dbd8fb-9lmcf" podUID="97e72214-c19b-401e-a6ed-f807439a575a" containerName="console" containerID="cri-o://61c0c4a3fbad4b2a0e5c7e4a90fb27ce06f56160b42754437c1f86210abdee54" gracePeriod=15
Jul 01 14:40:42.849743 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:42.849708    2558 generic.go:356] "Generic (PLEG): container finished" podID="97e72214-c19b-401e-a6ed-f807439a575a" containerID="61c0c4a3fbad4b2a0e5c7e4a90fb27ce06f56160b42754437c1f86210abdee54" exitCode=0
Jul 01 14:40:42.849889 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:42.849754    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f9dbd8fb-9lmcf" event={"ID":"97e72214-c19b-401e-a6ed-f807439a575a","Type":"ContainerDied","Data":"61c0c4a3fbad4b2a0e5c7e4a90fb27ce06f56160b42754437c1f86210abdee54"}
Jul 01 14:40:43.205465 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.205442    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:43.263276 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.263249    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-oauth-config\") pod \"97e72214-c19b-401e-a6ed-f807439a575a\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") "
Jul 01 14:40:43.263439 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.263307    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-serving-cert\") pod \"97e72214-c19b-401e-a6ed-f807439a575a\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") "
Jul 01 14:40:43.263439 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.263332    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-oauth-serving-cert\") pod \"97e72214-c19b-401e-a6ed-f807439a575a\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") "
Jul 01 14:40:43.263439 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.263404    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-service-ca\") pod \"97e72214-c19b-401e-a6ed-f807439a575a\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") "
Jul 01 14:40:43.263601 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.263441    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw45g\" (UniqueName: \"kubernetes.io/projected/97e72214-c19b-401e-a6ed-f807439a575a-kube-api-access-hw45g\") pod \"97e72214-c19b-401e-a6ed-f807439a575a\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") "
Jul 01 14:40:43.263601 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.263488    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-trusted-ca-bundle\") pod \"97e72214-c19b-401e-a6ed-f807439a575a\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") "
Jul 01 14:40:43.263601 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.263533    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-console-config\") pod \"97e72214-c19b-401e-a6ed-f807439a575a\" (UID: \"97e72214-c19b-401e-a6ed-f807439a575a\") "
Jul 01 14:40:43.263993 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.263906    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-service-ca" (OuterVolumeSpecName: "service-ca") pod "97e72214-c19b-401e-a6ed-f807439a575a" (UID: "97e72214-c19b-401e-a6ed-f807439a575a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:40:43.263993 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.263922    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97e72214-c19b-401e-a6ed-f807439a575a" (UID: "97e72214-c19b-401e-a6ed-f807439a575a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:40:43.263993 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.263983    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "97e72214-c19b-401e-a6ed-f807439a575a" (UID: "97e72214-c19b-401e-a6ed-f807439a575a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:40:43.264242 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.264031    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-console-config" (OuterVolumeSpecName: "console-config") pod "97e72214-c19b-401e-a6ed-f807439a575a" (UID: "97e72214-c19b-401e-a6ed-f807439a575a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:40:43.265676 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.265641    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97e72214-c19b-401e-a6ed-f807439a575a" (UID: "97e72214-c19b-401e-a6ed-f807439a575a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:40:43.265676 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.265648    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e72214-c19b-401e-a6ed-f807439a575a-kube-api-access-hw45g" (OuterVolumeSpecName: "kube-api-access-hw45g") pod "97e72214-c19b-401e-a6ed-f807439a575a" (UID: "97e72214-c19b-401e-a6ed-f807439a575a"). InnerVolumeSpecName "kube-api-access-hw45g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:40:43.265827 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.265760    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97e72214-c19b-401e-a6ed-f807439a575a" (UID: "97e72214-c19b-401e-a6ed-f807439a575a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:40:43.364745 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.364672    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-serving-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:40:43.364745 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.364710    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-oauth-serving-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:40:43.364745 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.364719    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:40:43.364745 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.364728    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw45g\" (UniqueName: \"kubernetes.io/projected/97e72214-c19b-401e-a6ed-f807439a575a-kube-api-access-hw45g\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:40:43.364745 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.364737    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-trusted-ca-bundle\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:40:43.364745 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.364745    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97e72214-c19b-401e-a6ed-f807439a575a-console-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:40:43.364992 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.364754    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97e72214-c19b-401e-a6ed-f807439a575a-console-oauth-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:40:43.854374 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.854343    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f9dbd8fb-9lmcf" event={"ID":"97e72214-c19b-401e-a6ed-f807439a575a","Type":"ContainerDied","Data":"685c8b3b5bc9c231055fd33e7ec716fb746cca55a3375d5274275db6549b48bc"}
Jul 01 14:40:43.854846 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.854375    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f9dbd8fb-9lmcf"
Jul 01 14:40:43.854846 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.854417    2558 scope.go:117] "RemoveContainer" containerID="61c0c4a3fbad4b2a0e5c7e4a90fb27ce06f56160b42754437c1f86210abdee54"
Jul 01 14:40:43.876958 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.876933    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9f9dbd8fb-9lmcf"]
Jul 01 14:40:43.879336 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:43.879316    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9f9dbd8fb-9lmcf"]
Jul 01 14:40:44.353951 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:44.353916    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97e72214-c19b-401e-a6ed-f807439a575a" path="/var/lib/kubelet/pods/97e72214-c19b-401e-a6ed-f807439a575a/volumes"
Jul 01 14:40:54.754191 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:54.754160    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-llw6c"
Jul 01 14:40:57.392858 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:57.392831    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:40:57.396627 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:40:57.396605    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-85b4d78d6f-768rp"
Jul 01 14:41:19.384218 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:19.384185    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:19.403155 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:19.403130    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:20.013316 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:20.013289    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:37.290288 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.290250    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-bff6f9c97-77rhh"]
Jul 01 14:41:37.290797 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.290770    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e72214-c19b-401e-a6ed-f807439a575a" containerName="console"
Jul 01 14:41:37.290797 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.290791    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e72214-c19b-401e-a6ed-f807439a575a" containerName="console"
Jul 01 14:41:37.290923 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.290874    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e72214-c19b-401e-a6ed-f807439a575a" containerName="console"
Jul 01 14:41:37.293402 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.293385    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.295748 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.295728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 01 14:41:37.295944 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.295916    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 01 14:41:37.296056 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.295916    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hnlb6\""
Jul 01 14:41:37.296056 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.296003    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 01 14:41:37.296056 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.296051    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 01 14:41:37.296451 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.296434    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 01 14:41:37.303290 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.303270    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 01 14:41:37.316208 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.316186    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bff6f9c97-77rhh"]
Jul 01 14:41:37.411007 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.410978    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb67b5f7-7bdb-4799-83a7-c130d15841f0-serving-certs-ca-bundle\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.411173 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.411039    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpz9\" (UniqueName: \"kubernetes.io/projected/cb67b5f7-7bdb-4799-83a7-c130d15841f0-kube-api-access-7xpz9\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.411173 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.411109    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb67b5f7-7bdb-4799-83a7-c130d15841f0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.411270 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.411180    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-telemeter-client-tls\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.411304 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.411294    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-federate-client-tls\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.411359 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.411348    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb67b5f7-7bdb-4799-83a7-c130d15841f0-metrics-client-ca\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.411441 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.411426    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.411482 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.411467    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-secret-telemeter-client\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.512465 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.512434    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb67b5f7-7bdb-4799-83a7-c130d15841f0-serving-certs-ca-bundle\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.512602 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.512471    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpz9\" (UniqueName: \"kubernetes.io/projected/cb67b5f7-7bdb-4799-83a7-c130d15841f0-kube-api-access-7xpz9\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.512602 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.512498    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb67b5f7-7bdb-4799-83a7-c130d15841f0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.512602 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.512536    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-telemeter-client-tls\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.512602 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.512580    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-federate-client-tls\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.512837 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.512611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb67b5f7-7bdb-4799-83a7-c130d15841f0-metrics-client-ca\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.512837 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.512657    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.512837 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.512712    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-secret-telemeter-client\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.513294 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.513262    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb67b5f7-7bdb-4799-83a7-c130d15841f0-serving-certs-ca-bundle\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.513421 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.513376    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb67b5f7-7bdb-4799-83a7-c130d15841f0-metrics-client-ca\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.513496 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.513425    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb67b5f7-7bdb-4799-83a7-c130d15841f0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.515149 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.515122    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-federate-client-tls\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.515262 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.515214    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-secret-telemeter-client\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.515393 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.515373    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.515525 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.515505    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cb67b5f7-7bdb-4799-83a7-c130d15841f0-telemeter-client-tls\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.524895 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.524867    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpz9\" (UniqueName: \"kubernetes.io/projected/cb67b5f7-7bdb-4799-83a7-c130d15841f0-kube-api-access-7xpz9\") pod \"telemeter-client-bff6f9c97-77rhh\" (UID: \"cb67b5f7-7bdb-4799-83a7-c130d15841f0\") " pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.605504 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.605483    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh"
Jul 01 14:41:37.744241 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:37.744218    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bff6f9c97-77rhh"]
Jul 01 14:41:37.746314 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:41:37.746291    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb67b5f7_7bdb_4799_83a7_c130d15841f0.slice/crio-f8b3bb5831f3ab66a2974b189de2992c778cf6a31813beb1b9a13c9044f0b2f1 WatchSource:0}: Error finding container f8b3bb5831f3ab66a2974b189de2992c778cf6a31813beb1b9a13c9044f0b2f1: Status 404 returned error can't find the container with id f8b3bb5831f3ab66a2974b189de2992c778cf6a31813beb1b9a13c9044f0b2f1
Jul 01 14:41:38.067441 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:38.067410    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh" event={"ID":"cb67b5f7-7bdb-4799-83a7-c130d15841f0","Type":"ContainerStarted","Data":"f8b3bb5831f3ab66a2974b189de2992c778cf6a31813beb1b9a13c9044f0b2f1"}
Jul 01 14:41:41.080206 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:41.080169    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh" event={"ID":"cb67b5f7-7bdb-4799-83a7-c130d15841f0","Type":"ContainerStarted","Data":"9ca8cb8e86231318757551a32f261cc9172466f28a1cf633c7892c20cdcaf496"}
Jul 01 14:41:41.080548 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:41.080224    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh" event={"ID":"cb67b5f7-7bdb-4799-83a7-c130d15841f0","Type":"ContainerStarted","Data":"6e945a35ddcf67a03162def550625e589baded238e8fb3cfe10416578dadafdc"}
Jul 01 14:41:41.080548 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:41.080245    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh" event={"ID":"cb67b5f7-7bdb-4799-83a7-c130d15841f0","Type":"ContainerStarted","Data":"943a68792001efd5714c347887b0bee169f0c9621c3f7507a5aa940c7f9af83a"}
Jul 01 14:41:41.110614 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:41.110563    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-bff6f9c97-77rhh" podStartSLOduration=1.249509568 podStartE2EDuration="4.110548081s" totalImagesPullingTime="2.861038513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:41:41.108181818 +0000 UTC m=+149.168761010" watchObservedRunningTime="2026-07-01 14:41:41.110548081 +0000 UTC m=+149.171127271"
Jul 01 14:41:59.026375 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:41:59.026340    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff88cbc7d-pr6jr"]
Jul 01 14:42:24.046662 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.046614    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6ff88cbc7d-pr6jr" podUID="13006537-9f80-448e-aa88-de938003c5de" containerName="console" containerID="cri-o://9f9273fea282ba420cbd4d9b3793f61fdf4cd4ec0925fc0a37e516af10d41d75" gracePeriod=15
Jul 01 14:42:24.250132 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.250099    2558 generic.go:356] "Generic (PLEG): container finished" podID="13006537-9f80-448e-aa88-de938003c5de" containerID="9f9273fea282ba420cbd4d9b3793f61fdf4cd4ec0925fc0a37e516af10d41d75" exitCode=0
Jul 01 14:42:24.250266 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.250174    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff88cbc7d-pr6jr" event={"ID":"13006537-9f80-448e-aa88-de938003c5de","Type":"ContainerDied","Data":"9f9273fea282ba420cbd4d9b3793f61fdf4cd4ec0925fc0a37e516af10d41d75"}
Jul 01 14:42:24.294995 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.294969    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:42:24.406906 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.406883    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-serving-cert\") pod \"13006537-9f80-448e-aa88-de938003c5de\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") "
Jul 01 14:42:24.407084 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.406936    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-trusted-ca-bundle\") pod \"13006537-9f80-448e-aa88-de938003c5de\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") "
Jul 01 14:42:24.407084 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.406962    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-oauth-config\") pod \"13006537-9f80-448e-aa88-de938003c5de\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") "
Jul 01 14:42:24.407084 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.407018    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-service-ca\") pod \"13006537-9f80-448e-aa88-de938003c5de\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") "
Jul 01 14:42:24.407243 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.407149    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hlks\" (UniqueName: \"kubernetes.io/projected/13006537-9f80-448e-aa88-de938003c5de-kube-api-access-8hlks\") pod \"13006537-9f80-448e-aa88-de938003c5de\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") "
Jul 01 14:42:24.407289 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.407253    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-console-config\") pod \"13006537-9f80-448e-aa88-de938003c5de\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") "
Jul 01 14:42:24.407343 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.407291    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-oauth-serving-cert\") pod \"13006537-9f80-448e-aa88-de938003c5de\" (UID: \"13006537-9f80-448e-aa88-de938003c5de\") "
Jul 01 14:42:24.407565 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.407496    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "13006537-9f80-448e-aa88-de938003c5de" (UID: "13006537-9f80-448e-aa88-de938003c5de"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:42:24.407644 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.407612    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-trusted-ca-bundle\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:42:24.407644 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.407620    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-console-config" (OuterVolumeSpecName: "console-config") pod "13006537-9f80-448e-aa88-de938003c5de" (UID: "13006537-9f80-448e-aa88-de938003c5de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:42:24.407786 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.407734    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "13006537-9f80-448e-aa88-de938003c5de" (UID: "13006537-9f80-448e-aa88-de938003c5de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:42:24.407786 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.407748    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-service-ca" (OuterVolumeSpecName: "service-ca") pod "13006537-9f80-448e-aa88-de938003c5de" (UID: "13006537-9f80-448e-aa88-de938003c5de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:42:24.409051 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.409032    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "13006537-9f80-448e-aa88-de938003c5de" (UID: "13006537-9f80-448e-aa88-de938003c5de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:42:24.409131 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.409081    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13006537-9f80-448e-aa88-de938003c5de-kube-api-access-8hlks" (OuterVolumeSpecName: "kube-api-access-8hlks") pod "13006537-9f80-448e-aa88-de938003c5de" (UID: "13006537-9f80-448e-aa88-de938003c5de"). InnerVolumeSpecName "kube-api-access-8hlks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:42:24.409199 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.409184    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "13006537-9f80-448e-aa88-de938003c5de" (UID: "13006537-9f80-448e-aa88-de938003c5de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:42:24.508678 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.508638    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-console-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:42:24.508678 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.508672    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-oauth-serving-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:42:24.508842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.508683    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-serving-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:42:24.508842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.508725    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13006537-9f80-448e-aa88-de938003c5de-console-oauth-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:42:24.508842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.508737    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13006537-9f80-448e-aa88-de938003c5de-service-ca\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:42:24.508842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:24.508748    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hlks\" (UniqueName: \"kubernetes.io/projected/13006537-9f80-448e-aa88-de938003c5de-kube-api-access-8hlks\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:42:25.255356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:25.255327    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff88cbc7d-pr6jr"
Jul 01 14:42:25.255356 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:25.255351    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff88cbc7d-pr6jr" event={"ID":"13006537-9f80-448e-aa88-de938003c5de","Type":"ContainerDied","Data":"170666771a983f146267a60c11058ecb21b906f81e3d56ec5cf5dbe60b04d84a"}
Jul 01 14:42:25.255840 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:25.255411    2558 scope.go:117] "RemoveContainer" containerID="9f9273fea282ba420cbd4d9b3793f61fdf4cd4ec0925fc0a37e516af10d41d75"
Jul 01 14:42:25.281282 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:25.281260    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff88cbc7d-pr6jr"]
Jul 01 14:42:25.285552 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:25.285531    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6ff88cbc7d-pr6jr"]
Jul 01 14:42:26.353770 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:42:26.353737    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13006537-9f80-448e-aa88-de938003c5de" path="/var/lib/kubelet/pods/13006537-9f80-448e-aa88-de938003c5de/volumes"
Jul 01 14:44:12.298502 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:12.298473    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:44:12.304049 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:12.304021    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:44:12.305730 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:12.305710    2558 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 14:44:27.889818 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:27.889788    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"]
Jul 01 14:44:27.891196 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:27.890102    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13006537-9f80-448e-aa88-de938003c5de" containerName="console"
Jul 01 14:44:27.891196 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:27.890113    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="13006537-9f80-448e-aa88-de938003c5de" containerName="console"
Jul 01 14:44:27.891196 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:27.890178    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="13006537-9f80-448e-aa88-de938003c5de" containerName="console"
Jul 01 14:44:27.892107 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:27.892089    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"
Jul 01 14:44:27.894634 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:27.894616    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 14:44:27.895584 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:27.895567    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 14:44:27.895635 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:27.895608    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 01 14:44:27.895892 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:27.895879    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-6v9gw\""
Jul 01 14:44:27.905241 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:27.905220    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"]
Jul 01 14:44:28.018661 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:28.018632    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4feb8bc-17bd-437c-b9b1-b9f2fff655c5-cert\") pod \"llmisvc-controller-manager-67d95fdc48-9dcl5\" (UID: \"f4feb8bc-17bd-437c-b9b1-b9f2fff655c5\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"
Jul 01 14:44:28.018838 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:28.018681    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrwlh\" (UniqueName: \"kubernetes.io/projected/f4feb8bc-17bd-437c-b9b1-b9f2fff655c5-kube-api-access-jrwlh\") pod \"llmisvc-controller-manager-67d95fdc48-9dcl5\" (UID: \"f4feb8bc-17bd-437c-b9b1-b9f2fff655c5\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"
Jul 01 14:44:28.119611 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:28.119579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4feb8bc-17bd-437c-b9b1-b9f2fff655c5-cert\") pod \"llmisvc-controller-manager-67d95fdc48-9dcl5\" (UID: \"f4feb8bc-17bd-437c-b9b1-b9f2fff655c5\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"
Jul 01 14:44:28.119784 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:28.119626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrwlh\" (UniqueName: \"kubernetes.io/projected/f4feb8bc-17bd-437c-b9b1-b9f2fff655c5-kube-api-access-jrwlh\") pod \"llmisvc-controller-manager-67d95fdc48-9dcl5\" (UID: \"f4feb8bc-17bd-437c-b9b1-b9f2fff655c5\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"
Jul 01 14:44:28.122189 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:28.122151    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4feb8bc-17bd-437c-b9b1-b9f2fff655c5-cert\") pod \"llmisvc-controller-manager-67d95fdc48-9dcl5\" (UID: \"f4feb8bc-17bd-437c-b9b1-b9f2fff655c5\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"
Jul 01 14:44:28.131355 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:28.131326    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrwlh\" (UniqueName: \"kubernetes.io/projected/f4feb8bc-17bd-437c-b9b1-b9f2fff655c5-kube-api-access-jrwlh\") pod \"llmisvc-controller-manager-67d95fdc48-9dcl5\" (UID: \"f4feb8bc-17bd-437c-b9b1-b9f2fff655c5\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"
Jul 01 14:44:28.202217 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:28.202164    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"
Jul 01 14:44:28.332777 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:28.332744    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"]
Jul 01 14:44:28.336294 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:44:28.336267    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf4feb8bc_17bd_437c_b9b1_b9f2fff655c5.slice/crio-d6767ae5b97f26e0d409cf0ff054169e54a767a2f77e494111dc84804f7ac3e3 WatchSource:0}: Error finding container d6767ae5b97f26e0d409cf0ff054169e54a767a2f77e494111dc84804f7ac3e3: Status 404 returned error can't find the container with id d6767ae5b97f26e0d409cf0ff054169e54a767a2f77e494111dc84804f7ac3e3
Jul 01 14:44:28.337907 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:28.337889    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 14:44:28.722540 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:28.722500    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5" event={"ID":"f4feb8bc-17bd-437c-b9b1-b9f2fff655c5","Type":"ContainerStarted","Data":"d6767ae5b97f26e0d409cf0ff054169e54a767a2f77e494111dc84804f7ac3e3"}
Jul 01 14:44:30.733507 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:30.733469    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5" event={"ID":"f4feb8bc-17bd-437c-b9b1-b9f2fff655c5","Type":"ContainerStarted","Data":"58215fb7d490974dadd9f1cfdecb5fa7f9eca8fbc6f7bd7286fd48c5fa7c6f17"}
Jul 01 14:44:30.733943 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:30.733555    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"
Jul 01 14:44:30.760754 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:44:30.760706    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5" podStartSLOduration=1.790699298 podStartE2EDuration="3.760674331s" totalImagesPullingTime="1.969975033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:44:30.759107284 +0000 UTC m=+318.819686474" watchObservedRunningTime="2026-07-01 14:44:30.760674331 +0000 UTC m=+318.821253521"
Jul 01 14:45:01.739255 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:01.739222    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-9dcl5"
Jul 01 14:45:53.790220 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.790189    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-4msh6"]
Jul 01 14:45:53.793508 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.793493    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4msh6"
Jul 01 14:45:53.800484 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.800459    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 14:45:53.800595 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.800483    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6874n\""
Jul 01 14:45:53.813869 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.813840    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c402ac9-ae39-4840-b0fe-65794297382a-model-cache\") pod \"s3-init-4msh6\" (UID: \"6c402ac9-ae39-4840-b0fe-65794297382a\") " pod="kserve/s3-init-4msh6"
Jul 01 14:45:53.813984 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.813962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxlw\" (UniqueName: \"kubernetes.io/projected/6c402ac9-ae39-4840-b0fe-65794297382a-kube-api-access-bnxlw\") pod \"s3-init-4msh6\" (UID: \"6c402ac9-ae39-4840-b0fe-65794297382a\") " pod="kserve/s3-init-4msh6"
Jul 01 14:45:53.814077 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.814059    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4msh6"]
Jul 01 14:45:53.915041 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.915014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxlw\" (UniqueName: \"kubernetes.io/projected/6c402ac9-ae39-4840-b0fe-65794297382a-kube-api-access-bnxlw\") pod \"s3-init-4msh6\" (UID: \"6c402ac9-ae39-4840-b0fe-65794297382a\") " pod="kserve/s3-init-4msh6"
Jul 01 14:45:53.915193 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.915056    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c402ac9-ae39-4840-b0fe-65794297382a-model-cache\") pod \"s3-init-4msh6\" (UID: \"6c402ac9-ae39-4840-b0fe-65794297382a\") " pod="kserve/s3-init-4msh6"
Jul 01 14:45:53.915399 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.915384    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c402ac9-ae39-4840-b0fe-65794297382a-model-cache\") pod \"s3-init-4msh6\" (UID: \"6c402ac9-ae39-4840-b0fe-65794297382a\") " pod="kserve/s3-init-4msh6"
Jul 01 14:45:53.932596 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:53.932578    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxlw\" (UniqueName: \"kubernetes.io/projected/6c402ac9-ae39-4840-b0fe-65794297382a-kube-api-access-bnxlw\") pod \"s3-init-4msh6\" (UID: \"6c402ac9-ae39-4840-b0fe-65794297382a\") " pod="kserve/s3-init-4msh6"
Jul 01 14:45:54.116425 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:54.116395    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4msh6"
Jul 01 14:45:54.245245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:54.245222    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4msh6"]
Jul 01 14:45:54.247919 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:45:54.247893    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c402ac9_ae39_4840_b0fe_65794297382a.slice/crio-491c7bfa535e360f4158581bc6f125769259e2845ca110b7552661bab8f60677 WatchSource:0}: Error finding container 491c7bfa535e360f4158581bc6f125769259e2845ca110b7552661bab8f60677: Status 404 returned error can't find the container with id 491c7bfa535e360f4158581bc6f125769259e2845ca110b7552661bab8f60677
Jul 01 14:45:55.057203 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:45:55.057176    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4msh6" event={"ID":"6c402ac9-ae39-4840-b0fe-65794297382a","Type":"ContainerStarted","Data":"491c7bfa535e360f4158581bc6f125769259e2845ca110b7552661bab8f60677"}
Jul 01 14:46:00.081284 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:46:00.081248    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4msh6" event={"ID":"6c402ac9-ae39-4840-b0fe-65794297382a","Type":"ContainerStarted","Data":"c541d1efd1b50fe2b3c84d43ca94607881f2782296b7c518b671992e4c92884c"}
Jul 01 14:46:01.087328 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:46:01.087294    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c402ac9-ae39-4840-b0fe-65794297382a" containerID="c541d1efd1b50fe2b3c84d43ca94607881f2782296b7c518b671992e4c92884c" exitCode=0
Jul 01 14:46:01.087779 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:46:01.087383    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4msh6" event={"ID":"6c402ac9-ae39-4840-b0fe-65794297382a","Type":"ContainerDied","Data":"c541d1efd1b50fe2b3c84d43ca94607881f2782296b7c518b671992e4c92884c"}
Jul 01 14:46:06.112254 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:46:06.112216    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4msh6" event={"ID":"6c402ac9-ae39-4840-b0fe-65794297382a","Type":"ContainerStarted","Data":"1e8596ae7c3042296006aa5a7bb78c51a6376cf2868e6e6d73e74bd81ebb86a6"}
Jul 01 14:46:06.129806 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:46:06.129749    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-4msh6" podStartSLOduration=1.728217639 podStartE2EDuration="13.129732698s" totalImagesPullingTime="10.005782467s" totalInitContainerRuntime="1.395732592s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:45:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:46:06.129063022 +0000 UTC m=+414.189642214" watchObservedRunningTime="2026-07-01 14:46:06.129732698 +0000 UTC m=+414.190311890"
Jul 01 14:47:11.372910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:11.372879    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c402ac9-ae39-4840-b0fe-65794297382a" containerID="1e8596ae7c3042296006aa5a7bb78c51a6376cf2868e6e6d73e74bd81ebb86a6" exitCode=0
Jul 01 14:47:11.373301 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:11.372953    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4msh6" event={"ID":"6c402ac9-ae39-4840-b0fe-65794297382a","Type":"ContainerDied","Data":"1e8596ae7c3042296006aa5a7bb78c51a6376cf2868e6e6d73e74bd81ebb86a6"}
Jul 01 14:47:12.503792 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:12.503770    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4msh6"
Jul 01 14:47:12.669721 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:12.669607    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnxlw\" (UniqueName: \"kubernetes.io/projected/6c402ac9-ae39-4840-b0fe-65794297382a-kube-api-access-bnxlw\") pod \"6c402ac9-ae39-4840-b0fe-65794297382a\" (UID: \"6c402ac9-ae39-4840-b0fe-65794297382a\") "
Jul 01 14:47:12.669860 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:12.669721    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c402ac9-ae39-4840-b0fe-65794297382a-model-cache\") pod \"6c402ac9-ae39-4840-b0fe-65794297382a\" (UID: \"6c402ac9-ae39-4840-b0fe-65794297382a\") "
Jul 01 14:47:12.671808 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:12.671781    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c402ac9-ae39-4840-b0fe-65794297382a-kube-api-access-bnxlw" (OuterVolumeSpecName: "kube-api-access-bnxlw") pod "6c402ac9-ae39-4840-b0fe-65794297382a" (UID: "6c402ac9-ae39-4840-b0fe-65794297382a"). InnerVolumeSpecName "kube-api-access-bnxlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:47:12.688087 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:12.688038    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c402ac9-ae39-4840-b0fe-65794297382a-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c402ac9-ae39-4840-b0fe-65794297382a" (UID: "6c402ac9-ae39-4840-b0fe-65794297382a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:47:12.770671 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:12.770641    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxlw\" (UniqueName: \"kubernetes.io/projected/6c402ac9-ae39-4840-b0fe-65794297382a-kube-api-access-bnxlw\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:47:12.770671 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:12.770664    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c402ac9-ae39-4840-b0fe-65794297382a-model-cache\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:47:13.381520 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:13.381486    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4msh6" event={"ID":"6c402ac9-ae39-4840-b0fe-65794297382a","Type":"ContainerDied","Data":"491c7bfa535e360f4158581bc6f125769259e2845ca110b7552661bab8f60677"}
Jul 01 14:47:13.381520 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:13.381522    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="491c7bfa535e360f4158581bc6f125769259e2845ca110b7552661bab8f60677"
Jul 01 14:47:13.381520 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:13.381521    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4msh6"
Jul 01 14:47:25.071556 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.071523    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"]
Jul 01 14:47:25.071938 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.071875    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c402ac9-ae39-4840-b0fe-65794297382a" containerName="s3-init"
Jul 01 14:47:25.071938 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.071887    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c402ac9-ae39-4840-b0fe-65794297382a" containerName="s3-init"
Jul 01 14:47:25.071938 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.071905    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c402ac9-ae39-4840-b0fe-65794297382a" containerName="download-hf-model"
Jul 01 14:47:25.071938 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.071911    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c402ac9-ae39-4840-b0fe-65794297382a" containerName="download-hf-model"
Jul 01 14:47:25.072065 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.071964    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c402ac9-ae39-4840-b0fe-65794297382a" containerName="s3-init"
Jul 01 14:47:25.075292 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.075274    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.077389 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.077371    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2d7fd-predictor-serving-cert\""
Jul 01 14:47:25.077616 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.077597    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-n79zh\""
Jul 01 14:47:25.077716 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.077604    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 14:47:25.077716 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.077648    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2d7fd-kube-rbac-proxy-sar-config\""
Jul 01 14:47:25.078385 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.078367    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 14:47:25.078487 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.078421    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 14:47:25.078487 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.078448    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 14:47:25.087351 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.087332    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"]
Jul 01 14:47:25.272206 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.272170    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b0e650-2b56-4cbe-bbc5-d23340095e08-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.272382 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.272217    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.272382 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.272240    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fff2f\" (UniqueName: \"kubernetes.io/projected/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-api-access-fff2f\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.272382 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.272292    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.272382 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.272368    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.372988 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.372910    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b0e650-2b56-4cbe-bbc5-d23340095e08-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.372988 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.372957    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.372988 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.372976    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fff2f\" (UniqueName: \"kubernetes.io/projected/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-api-access-fff2f\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.372988 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.372993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.373259 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.373043    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.373504 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.373465    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.373710 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.373670    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.373790 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.373671    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.375372 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.375354    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b0e650-2b56-4cbe-bbc5-d23340095e08-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.382468 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.382449    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fff2f\" (UniqueName: \"kubernetes.io/projected/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-api-access-fff2f\") pod \"isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.385271 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.385252    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:25.523775 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:25.523748    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"]
Jul 01 14:47:25.525764 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:47:25.525737    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b0e650_2b56_4cbe_bbc5_d23340095e08.slice/crio-daf4f2d785360a8f246cbd272bd007b34dc8babb970bc60a44f2866de56e27c1 WatchSource:0}: Error finding container daf4f2d785360a8f246cbd272bd007b34dc8babb970bc60a44f2866de56e27c1: Status 404 returned error can't find the container with id daf4f2d785360a8f246cbd272bd007b34dc8babb970bc60a44f2866de56e27c1
Jul 01 14:47:26.435678 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:26.435640    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" event={"ID":"d6b0e650-2b56-4cbe-bbc5-d23340095e08","Type":"ContainerStarted","Data":"66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca"}
Jul 01 14:47:26.436132 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:26.435739    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" event={"ID":"d6b0e650-2b56-4cbe-bbc5-d23340095e08","Type":"ContainerStarted","Data":"daf4f2d785360a8f246cbd272bd007b34dc8babb970bc60a44f2866de56e27c1"}
Jul 01 14:47:29.450235 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:29.450200    2558 generic.go:356] "Generic (PLEG): container finished" podID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerID="66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca" exitCode=0
Jul 01 14:47:29.450560 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:29.450262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" event={"ID":"d6b0e650-2b56-4cbe-bbc5-d23340095e08","Type":"ContainerDied","Data":"66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca"}
Jul 01 14:47:45.536668 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:45.536586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" event={"ID":"d6b0e650-2b56-4cbe-bbc5-d23340095e08","Type":"ContainerStarted","Data":"9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f"}
Jul 01 14:47:48.551422 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:48.551386    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" event={"ID":"d6b0e650-2b56-4cbe-bbc5-d23340095e08","Type":"ContainerStarted","Data":"3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93"}
Jul 01 14:47:51.566639 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:51.566611    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" event={"ID":"d6b0e650-2b56-4cbe-bbc5-d23340095e08","Type":"ContainerStarted","Data":"108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b"}
Jul 01 14:47:51.567106 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:51.566858    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:51.567106 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:51.566998    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:51.568301 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:51.568276    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:47:51.589555 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:51.589509    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podStartSLOduration=1.31492104 podStartE2EDuration="26.58949322s" totalImagesPullingTime="21.572472399s" totalInitContainerRuntime="3.702099781s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:47:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:47:51.588223247 +0000 UTC m=+519.648802438" watchObservedRunningTime="2026-07-01 14:47:51.58949322 +0000 UTC m=+519.650072410"
Jul 01 14:47:52.570441 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:52.570405    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:52.570947 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:52.570572    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:47:52.571570 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:52.571538    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:47:52.574391 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:52.574373    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:47:53.575126 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:53.575085    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:47:53.575516 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:53.575329    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:47:54.578969 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:54.578927    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:47:54.579379 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:47:54.579166    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:48:04.579682 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:04.579638    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:48:04.580171 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:04.580148    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:48:14.579377 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:14.579335    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:48:14.579905 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:14.579855    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:48:24.579869 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:24.579817    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:48:24.580345 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:24.580299    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:48:34.579380 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:34.579328    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:48:34.579907 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:34.579881    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:48:44.578895 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:44.578841    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:48:44.579310 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:44.579288    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:48:54.579564 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:54.579520    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:48:54.580013 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:48:54.579988    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:49:04.580365 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:04.580333    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:49:04.580907 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:04.580481    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:49:10.210343 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.210309    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"]
Jul 01 14:49:10.210750 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.210607    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" containerID="cri-o://9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f" gracePeriod=30
Jul 01 14:49:10.210750 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.210633    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" containerID="cri-o://108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b" gracePeriod=30
Jul 01 14:49:10.210750 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.210647    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kube-rbac-proxy" containerID="cri-o://3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93" gracePeriod=30
Jul 01 14:49:10.232792 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.232771    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"]
Jul 01 14:49:10.236365 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.236338    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.239003 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.238983    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-63a63-predictor-serving-cert\""
Jul 01 14:49:10.239290 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.239267    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-63a63-kube-rbac-proxy-sar-config\""
Jul 01 14:49:10.248390 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.248367    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"]
Jul 01 14:49:10.305203 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.305171    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-cabundle-cert\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.305352 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.305218    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c2dede-364e-423f-938e-94b314e7f815-proxy-tls\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.305352 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.305235    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjqv\" (UniqueName: \"kubernetes.io/projected/22c2dede-364e-423f-938e-94b314e7f815-kube-api-access-bdjqv\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.305352 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.305254    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.305352 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.305334    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c2dede-364e-423f-938e-94b314e7f815-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.317422 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.317398    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"]
Jul 01 14:49:10.321156 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.321133    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.324245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.324225    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-63a63-predictor-serving-cert\""
Jul 01 14:49:10.324559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.324541    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-63a63-kube-rbac-proxy-sar-config\""
Jul 01 14:49:10.336469 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.336450    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"]
Jul 01 14:49:10.405829 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.405801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c2dede-364e-423f-938e-94b314e7f815-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.405997 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.405841    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45794eea-ae6e-4638-8ea4-db648638cf41-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.405997 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.405861    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stxgj\" (UniqueName: \"kubernetes.io/projected/45794eea-ae6e-4638-8ea4-db648638cf41-kube-api-access-stxgj\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.405997 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.405916    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45794eea-ae6e-4638-8ea4-db648638cf41-proxy-tls\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.405997 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.405946    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-cabundle-cert\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.405997 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.405993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-cabundle-cert\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.406266 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.406016    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.406266 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.406045    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c2dede-364e-423f-938e-94b314e7f815-proxy-tls\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.406266 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.406062    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjqv\" (UniqueName: \"kubernetes.io/projected/22c2dede-364e-423f-938e-94b314e7f815-kube-api-access-bdjqv\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.406266 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.406079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.406266 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.406234    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c2dede-364e-423f-938e-94b314e7f815-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.406580 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.406557    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-cabundle-cert\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.406727 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.406653    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.408457 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.408437    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c2dede-364e-423f-938e-94b314e7f815-proxy-tls\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.418350 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.418330    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjqv\" (UniqueName: \"kubernetes.io/projected/22c2dede-364e-423f-938e-94b314e7f815-kube-api-access-bdjqv\") pod \"isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.506660 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.506589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.506660 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.506642    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45794eea-ae6e-4638-8ea4-db648638cf41-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.506660 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.506661    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stxgj\" (UniqueName: \"kubernetes.io/projected/45794eea-ae6e-4638-8ea4-db648638cf41-kube-api-access-stxgj\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.506960 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.506712    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45794eea-ae6e-4638-8ea4-db648638cf41-proxy-tls\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.506960 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.506739    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-cabundle-cert\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.507112 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.507089    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45794eea-ae6e-4638-8ea4-db648638cf41-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.507279 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.507260    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-cabundle-cert\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.507341 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.507305    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.509036 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.509014    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45794eea-ae6e-4638-8ea4-db648638cf41-proxy-tls\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.518140 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.518118    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stxgj\" (UniqueName: \"kubernetes.io/projected/45794eea-ae6e-4638-8ea4-db648638cf41-kube-api-access-stxgj\") pod \"isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.547111 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.547090    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:10.632111 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.632079    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:10.683099 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.683066    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"]
Jul 01 14:49:10.686717 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:49:10.686670    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c2dede_364e_423f_938e_94b314e7f815.slice/crio-8aaeee3e812e10c4a47eedfe4e2a990650daa4b664bc30d5af7f4924403d0327 WatchSource:0}: Error finding container 8aaeee3e812e10c4a47eedfe4e2a990650daa4b664bc30d5af7f4924403d0327: Status 404 returned error can't find the container with id 8aaeee3e812e10c4a47eedfe4e2a990650daa4b664bc30d5af7f4924403d0327
Jul 01 14:49:10.776538 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.776516    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"]
Jul 01 14:49:10.778861 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:49:10.778832    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45794eea_ae6e_4638_8ea4_db648638cf41.slice/crio-5c53304c985ff1bf0d0295f4007c1b7517ba536fa68d03605d01ce6b34ca6091 WatchSource:0}: Error finding container 5c53304c985ff1bf0d0295f4007c1b7517ba536fa68d03605d01ce6b34ca6091: Status 404 returned error can't find the container with id 5c53304c985ff1bf0d0295f4007c1b7517ba536fa68d03605d01ce6b34ca6091
Jul 01 14:49:10.897466 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.897421    2558 generic.go:356] "Generic (PLEG): container finished" podID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerID="3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93" exitCode=2
Jul 01 14:49:10.897643 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.897550    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" event={"ID":"d6b0e650-2b56-4cbe-bbc5-d23340095e08","Type":"ContainerDied","Data":"3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93"}
Jul 01 14:49:10.899726 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.899657    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" event={"ID":"45794eea-ae6e-4638-8ea4-db648638cf41","Type":"ContainerStarted","Data":"f117d8094b457788167b2bfd7c65c0dcd4e116f49083865e284e17378aff6d08"}
Jul 01 14:49:10.899880 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.899778    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" event={"ID":"45794eea-ae6e-4638-8ea4-db648638cf41","Type":"ContainerStarted","Data":"5c53304c985ff1bf0d0295f4007c1b7517ba536fa68d03605d01ce6b34ca6091"}
Jul 01 14:49:10.901508 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.901483    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" event={"ID":"22c2dede-364e-423f-938e-94b314e7f815","Type":"ContainerStarted","Data":"d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323"}
Jul 01 14:49:10.901621 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:10.901527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" event={"ID":"22c2dede-364e-423f-938e-94b314e7f815","Type":"ContainerStarted","Data":"8aaeee3e812e10c4a47eedfe4e2a990650daa4b664bc30d5af7f4924403d0327"}
Jul 01 14:49:12.336363 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:12.336336    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:49:12.337568 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:12.337546    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:49:12.571262 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:12.571224    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 01 14:49:14.579177 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:14.579145    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:49:14.579482 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:14.579446    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:49:14.920590 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:14.920564    2558 generic.go:356] "Generic (PLEG): container finished" podID="22c2dede-364e-423f-938e-94b314e7f815" containerID="d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323" exitCode=0
Jul 01 14:49:14.920765 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:14.920635    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" event={"ID":"22c2dede-364e-423f-938e-94b314e7f815","Type":"ContainerDied","Data":"d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323"}
Jul 01 14:49:14.922979 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:14.922960    2558 generic.go:356] "Generic (PLEG): container finished" podID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerID="9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f" exitCode=0
Jul 01 14:49:14.923084 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:14.923024    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" event={"ID":"d6b0e650-2b56-4cbe-bbc5-d23340095e08","Type":"ContainerDied","Data":"9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f"}
Jul 01 14:49:14.924470 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:14.924445    2558 generic.go:356] "Generic (PLEG): container finished" podID="45794eea-ae6e-4638-8ea4-db648638cf41" containerID="f117d8094b457788167b2bfd7c65c0dcd4e116f49083865e284e17378aff6d08" exitCode=0
Jul 01 14:49:14.924549 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:14.924515    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" event={"ID":"45794eea-ae6e-4638-8ea4-db648638cf41","Type":"ContainerDied","Data":"f117d8094b457788167b2bfd7c65c0dcd4e116f49083865e284e17378aff6d08"}
Jul 01 14:49:15.941973 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:15.941936    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" event={"ID":"22c2dede-364e-423f-938e-94b314e7f815","Type":"ContainerStarted","Data":"737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0"}
Jul 01 14:49:15.942388 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:15.942027    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" event={"ID":"22c2dede-364e-423f-938e-94b314e7f815","Type":"ContainerStarted","Data":"f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92"}
Jul 01 14:49:15.942388 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:15.942346    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:15.942986 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:15.942961    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:15.944193 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:15.944163    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:49:15.966369 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:15.966306    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podStartSLOduration=2.22464557 podStartE2EDuration="5.966288958s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.741643388s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:49:15.964006121 +0000 UTC m=+604.024585317" watchObservedRunningTime="2026-07-01 14:49:15.966288958 +0000 UTC m=+604.026868148"
Jul 01 14:49:16.948345 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:16.947927    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:49:17.570916 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:17.570876    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 01 14:49:17.953151 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:17.953063    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:49:22.571080 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:22.571037    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 01 14:49:22.571511 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:22.571171    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:49:22.958476 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:22.958396    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:49:22.959125 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:22.959089    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:49:24.579676 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:24.579624    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:49:24.580153 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:24.580010    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:49:27.571010 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:27.570965    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 01 14:49:32.570994 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:32.570955    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 01 14:49:32.959087 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:32.959008    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:49:33.025616 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:33.025579    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" event={"ID":"45794eea-ae6e-4638-8ea4-db648638cf41","Type":"ContainerStarted","Data":"e94f914c60c959d569b2d7f48ca979396041c48192b70c8d8e2ba33e253d974d"}
Jul 01 14:49:33.025806 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:33.025657    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" event={"ID":"45794eea-ae6e-4638-8ea4-db648638cf41","Type":"ContainerStarted","Data":"ef12ee51db9f0d6dacc043b19106d79ec4c799e12cface4725e4f604c7aeb81b"}
Jul 01 14:49:33.025892 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:33.025874    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:33.054231 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:33.054185    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podStartSLOduration=1.593947303 podStartE2EDuration="23.054170896s" totalImagesPullingTime="17.658588299s" totalInitContainerRuntime="3.801635294s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:49:33.052854462 +0000 UTC m=+621.113433655" watchObservedRunningTime="2026-07-01 14:49:33.054170896 +0000 UTC m=+621.114750086"
Jul 01 14:49:34.030715 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:34.030675    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:34.031869 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:34.031842    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 14:49:34.578910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:34.578861    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 14:49:34.579075 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:34.579019    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:49:34.579128 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:34.579105    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:49:34.579232 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:34.579220    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:49:35.035053 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:35.034949    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 14:49:37.571240 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:37.571205    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 01 14:49:40.039705 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.039655    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:49:40.040233 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.040206    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 14:49:40.909451 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.909422    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:49:40.989857 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.989825    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-rbac-proxy-sar-config\") pod \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") "
Jul 01 14:49:40.990001 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.989891    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kserve-provision-location\") pod \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") "
Jul 01 14:49:40.990001 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.989921    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b0e650-2b56-4cbe-bbc5-d23340095e08-proxy-tls\") pod \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") "
Jul 01 14:49:40.990001 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.989982    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-cabundle-cert\") pod \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") "
Jul 01 14:49:40.990174 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.990009    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fff2f\" (UniqueName: \"kubernetes.io/projected/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-api-access-fff2f\") pod \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\" (UID: \"d6b0e650-2b56-4cbe-bbc5-d23340095e08\") "
Jul 01 14:49:40.990298 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.990266    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6b0e650-2b56-4cbe-bbc5-d23340095e08" (UID: "d6b0e650-2b56-4cbe-bbc5-d23340095e08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:49:40.990367 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.990291    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6b0e650-2b56-4cbe-bbc5-d23340095e08" (UID: "d6b0e650-2b56-4cbe-bbc5-d23340095e08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:49:40.990367 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.990333    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6b0e650-2b56-4cbe-bbc5-d23340095e08" (UID: "d6b0e650-2b56-4cbe-bbc5-d23340095e08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:49:40.992262 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.992221    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-api-access-fff2f" (OuterVolumeSpecName: "kube-api-access-fff2f") pod "d6b0e650-2b56-4cbe-bbc5-d23340095e08" (UID: "d6b0e650-2b56-4cbe-bbc5-d23340095e08"). InnerVolumeSpecName "kube-api-access-fff2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:49:40.992370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:40.992287    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b0e650-2b56-4cbe-bbc5-d23340095e08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6b0e650-2b56-4cbe-bbc5-d23340095e08" (UID: "d6b0e650-2b56-4cbe-bbc5-d23340095e08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:49:41.065488 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.065410    2558 generic.go:356] "Generic (PLEG): container finished" podID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerID="108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b" exitCode=0
Jul 01 14:49:41.065488 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.065466    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"
Jul 01 14:49:41.065488 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.065475    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" event={"ID":"d6b0e650-2b56-4cbe-bbc5-d23340095e08","Type":"ContainerDied","Data":"108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b"}
Jul 01 14:49:41.065962 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.065581    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp" event={"ID":"d6b0e650-2b56-4cbe-bbc5-d23340095e08","Type":"ContainerDied","Data":"daf4f2d785360a8f246cbd272bd007b34dc8babb970bc60a44f2866de56e27c1"}
Jul 01 14:49:41.065962 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.065626    2558 scope.go:117] "RemoveContainer" containerID="108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b"
Jul 01 14:49:41.074058 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.074040    2558 scope.go:117] "RemoveContainer" containerID="3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93"
Jul 01 14:49:41.081324 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.081310    2558 scope.go:117] "RemoveContainer" containerID="9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f"
Jul 01 14:49:41.089753 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.089204    2558 scope.go:117] "RemoveContainer" containerID="66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca"
Jul 01 14:49:41.090672 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.090652    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:49:41.090672 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.090673    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:49:41.091217 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.090707    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b0e650-2b56-4cbe-bbc5-d23340095e08-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:49:41.091217 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.090724    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6b0e650-2b56-4cbe-bbc5-d23340095e08-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:49:41.091217 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.090735    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fff2f\" (UniqueName: \"kubernetes.io/projected/d6b0e650-2b56-4cbe-bbc5-d23340095e08-kube-api-access-fff2f\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:49:41.093178 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.093151    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"]
Jul 01 14:49:41.097146 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.097123    2558 scope.go:117] "RemoveContainer" containerID="108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b"
Jul 01 14:49:41.097488 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:49:41.097454    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b\": container with ID starting with 108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b not found: ID does not exist" containerID="108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b"
Jul 01 14:49:41.097581 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.097495    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b"} err="failed to get container status \"108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b\": rpc error: code = NotFound desc = could not find container \"108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b\": container with ID starting with 108874b1e57ccf9be10a0c7a0555d6a5df10f7b8adc9c6a8e3ecf4e21fb65e3b not found: ID does not exist"
Jul 01 14:49:41.097581 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.097534    2558 scope.go:117] "RemoveContainer" containerID="3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93"
Jul 01 14:49:41.097846 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:49:41.097821    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93\": container with ID starting with 3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93 not found: ID does not exist" containerID="3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93"
Jul 01 14:49:41.097943 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.097850    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93"} err="failed to get container status \"3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93\": rpc error: code = NotFound desc = could not find container \"3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93\": container with ID starting with 3f40ddced4be77cc3328ab538ff7e0b560593a14175a1091742b8e749c4a9b93 not found: ID does not exist"
Jul 01 14:49:41.097943 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.097866    2558 scope.go:117] "RemoveContainer" containerID="9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f"
Jul 01 14:49:41.098163 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:49:41.098139    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f\": container with ID starting with 9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f not found: ID does not exist" containerID="9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f"
Jul 01 14:49:41.098229 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.098170    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f"} err="failed to get container status \"9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f\": rpc error: code = NotFound desc = could not find container \"9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f\": container with ID starting with 9e402efa4ca880be134ccae33111242ce6b8ed7c824241e33fd43cc9b587ba9f not found: ID does not exist"
Jul 01 14:49:41.098229 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.098189    2558 scope.go:117] "RemoveContainer" containerID="66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca"
Jul 01 14:49:41.098532 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:49:41.098505    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca\": container with ID starting with 66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca not found: ID does not exist" containerID="66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca"
Jul 01 14:49:41.098595 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.098529    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca"} err="failed to get container status \"66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca\": rpc error: code = NotFound desc = could not find container \"66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca\": container with ID starting with 66c1fe1b5edf58749fd413068bea107499b69ebe6dab1e924684305eede09fca not found: ID does not exist"
Jul 01 14:49:41.100643 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:41.100624    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2d7fd-predictor-68d48f864b-tgzlp"]
Jul 01 14:49:42.353200 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:42.353173    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" path="/var/lib/kubelet/pods/d6b0e650-2b56-4cbe-bbc5-d23340095e08/volumes"
Jul 01 14:49:42.959355 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:42.959301    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:49:50.040185 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:50.040146    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 14:49:52.959870 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:49:52.959829    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:50:00.040123 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:50:00.040082    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 14:50:02.959632 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:50:02.959597    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:50:10.040893 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:50:10.040853    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 14:50:12.959287 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:50:12.959247    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:50:20.040833 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:50:20.040789    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 14:50:22.959599 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:50:22.959559    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:50:30.041125 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:50:30.041081    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 14:50:32.959925 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:50:32.959886    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:50:40.041634 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:50:40.041607    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:51:00.606962 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.606929    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"]
Jul 01 14:51:00.607375 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607290    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent"
Jul 01 14:51:00.607375 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607303    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent"
Jul 01 14:51:00.607375 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607319    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kube-rbac-proxy"
Jul 01 14:51:00.607375 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607325    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kube-rbac-proxy"
Jul 01 14:51:00.607375 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607339    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="storage-initializer"
Jul 01 14:51:00.607375 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607345    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="storage-initializer"
Jul 01 14:51:00.607375 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607355    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container"
Jul 01 14:51:00.607375 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607361    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container"
Jul 01 14:51:00.607629 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607431    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="agent"
Jul 01 14:51:00.607629 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607445    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kserve-container"
Jul 01 14:51:00.607629 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.607453    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b0e650-2b56-4cbe-bbc5-d23340095e08" containerName="kube-rbac-proxy"
Jul 01 14:51:00.609575 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.609560    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.612532 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.612508    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-02737-predictor-serving-cert\""
Jul 01 14:51:00.612798 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.612783    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-02737-kube-rbac-proxy-sar-config\""
Jul 01 14:51:00.622288 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.622268    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"]
Jul 01 14:51:00.626152 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.626128    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"]
Jul 01 14:51:00.626350 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.626330    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" containerID="cri-o://f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92" gracePeriod=30
Jul 01 14:51:00.626428 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.626395    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kube-rbac-proxy" containerID="cri-o://737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0" gracePeriod=30
Jul 01 14:51:00.668577 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.668552    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.668739 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.668595    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnqz\" (UniqueName: \"kubernetes.io/projected/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-api-access-ltnqz\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.668739 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.668623    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b056f5a5-627b-4263-8487-0bff33f86e6e-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.668864 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.668734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.668922 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.668863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b056f5a5-627b-4263-8487-0bff33f86e6e-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.758081 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.758053    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"]
Jul 01 14:51:00.758328 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.758300    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container" containerID="cri-o://ef12ee51db9f0d6dacc043b19106d79ec4c799e12cface4725e4f604c7aeb81b" gracePeriod=30
Jul 01 14:51:00.758526 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.758347    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kube-rbac-proxy" containerID="cri-o://e94f914c60c959d569b2d7f48ca979396041c48192b70c8d8e2ba33e253d974d" gracePeriod=30
Jul 01 14:51:00.769440 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.769421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.769513 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.769468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnqz\" (UniqueName: \"kubernetes.io/projected/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-api-access-ltnqz\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.769513 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.769490    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b056f5a5-627b-4263-8487-0bff33f86e6e-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.769657 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.769619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.769807 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.769768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b056f5a5-627b-4263-8487-0bff33f86e6e-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.770164 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.770142    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.770252 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.770162    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b056f5a5-627b-4263-8487-0bff33f86e6e-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.770252 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.770212    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.772142 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.772122    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b056f5a5-627b-4263-8487-0bff33f86e6e-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.782141 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.782113    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnqz\" (UniqueName: \"kubernetes.io/projected/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-api-access-ltnqz\") pod \"isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.789953 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.789934    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"]
Jul 01 14:51:00.792905 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.792890    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.795822 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.795799    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-02737-kube-rbac-proxy-sar-config\""
Jul 01 14:51:00.795933 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.795804    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-02737-predictor-serving-cert\""
Jul 01 14:51:00.811027 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.811009    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"]
Jul 01 14:51:00.870990 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.870929    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgsv6\" (UniqueName: \"kubernetes.io/projected/d788da9c-cd56-43aa-837b-aee97423de11-kube-api-access-pgsv6\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.870990 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.870963    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.870990 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.870984    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.871148 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.871063    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d788da9c-cd56-43aa-837b-aee97423de11-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.871148 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.871097    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d788da9c-cd56-43aa-837b-aee97423de11-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.919481 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.919452    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:00.971867 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.971832    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgsv6\" (UniqueName: \"kubernetes.io/projected/d788da9c-cd56-43aa-837b-aee97423de11-kube-api-access-pgsv6\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.972006 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.971898    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.972006 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.971930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.972006 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.971984    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d788da9c-cd56-43aa-837b-aee97423de11-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.972182 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.972014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d788da9c-cd56-43aa-837b-aee97423de11-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.972525 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.972497    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d788da9c-cd56-43aa-837b-aee97423de11-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.972627 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.972610    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.973073 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.973048    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.974552 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.974527    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d788da9c-cd56-43aa-837b-aee97423de11-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:00.981609 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:00.981582    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgsv6\" (UniqueName: \"kubernetes.io/projected/d788da9c-cd56-43aa-837b-aee97423de11-kube-api-access-pgsv6\") pod \"isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:01.051495 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.051471    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"]
Jul 01 14:51:01.053647 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:51:01.053620    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb056f5a5_627b_4263_8487_0bff33f86e6e.slice/crio-cccf634ec20822620caf1192eff7730a3333bcfe05f3568a3f611d256a081da8 WatchSource:0}: Error finding container cccf634ec20822620caf1192eff7730a3333bcfe05f3568a3f611d256a081da8: Status 404 returned error can't find the container with id cccf634ec20822620caf1192eff7730a3333bcfe05f3568a3f611d256a081da8
Jul 01 14:51:01.103423 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.103403    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:01.240555 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.240429    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"]
Jul 01 14:51:01.243215 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:51:01.243186    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd788da9c_cd56_43aa_837b_aee97423de11.slice/crio-e41806bccc31d6e0cf815f2da08f2e1d5d5345444ff0c46e35bae75bc0920f62 WatchSource:0}: Error finding container e41806bccc31d6e0cf815f2da08f2e1d5d5345444ff0c46e35bae75bc0920f62: Status 404 returned error can't find the container with id e41806bccc31d6e0cf815f2da08f2e1d5d5345444ff0c46e35bae75bc0920f62
Jul 01 14:51:01.406463 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.406371    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" event={"ID":"b056f5a5-627b-4263-8487-0bff33f86e6e","Type":"ContainerStarted","Data":"05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3"}
Jul 01 14:51:01.406463 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.406445    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" event={"ID":"b056f5a5-627b-4263-8487-0bff33f86e6e","Type":"ContainerStarted","Data":"cccf634ec20822620caf1192eff7730a3333bcfe05f3568a3f611d256a081da8"}
Jul 01 14:51:01.408883 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.408859    2558 generic.go:356] "Generic (PLEG): container finished" podID="45794eea-ae6e-4638-8ea4-db648638cf41" containerID="e94f914c60c959d569b2d7f48ca979396041c48192b70c8d8e2ba33e253d974d" exitCode=2
Jul 01 14:51:01.409022 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.408942    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" event={"ID":"45794eea-ae6e-4638-8ea4-db648638cf41","Type":"ContainerDied","Data":"e94f914c60c959d569b2d7f48ca979396041c48192b70c8d8e2ba33e253d974d"}
Jul 01 14:51:01.411334 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.411302    2558 generic.go:356] "Generic (PLEG): container finished" podID="22c2dede-364e-423f-938e-94b314e7f815" containerID="737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0" exitCode=2
Jul 01 14:51:01.411439 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.411377    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" event={"ID":"22c2dede-364e-423f-938e-94b314e7f815","Type":"ContainerDied","Data":"737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0"}
Jul 01 14:51:01.413137 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.413108    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" event={"ID":"d788da9c-cd56-43aa-837b-aee97423de11","Type":"ContainerStarted","Data":"917b2a4ab62d714485fdb224f109cd1edd7d29dba3e41e5dc36d76b001f3af57"}
Jul 01 14:51:01.413225 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:01.413167    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" event={"ID":"d788da9c-cd56-43aa-837b-aee97423de11","Type":"ContainerStarted","Data":"e41806bccc31d6e0cf815f2da08f2e1d5d5345444ff0c46e35bae75bc0920f62"}
Jul 01 14:51:02.953975 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:02.953935    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 01 14:51:02.959738 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:02.959711    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 14:51:04.429828 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.429795    2558 generic.go:356] "Generic (PLEG): container finished" podID="45794eea-ae6e-4638-8ea4-db648638cf41" containerID="ef12ee51db9f0d6dacc043b19106d79ec4c799e12cface4725e4f604c7aeb81b" exitCode=0
Jul 01 14:51:04.430116 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.429860    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" event={"ID":"45794eea-ae6e-4638-8ea4-db648638cf41","Type":"ContainerDied","Data":"ef12ee51db9f0d6dacc043b19106d79ec4c799e12cface4725e4f604c7aeb81b"}
Jul 01 14:51:04.504646 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.504617    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:51:04.605561 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.605536    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stxgj\" (UniqueName: \"kubernetes.io/projected/45794eea-ae6e-4638-8ea4-db648638cf41-kube-api-access-stxgj\") pod \"45794eea-ae6e-4638-8ea4-db648638cf41\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") "
Jul 01 14:51:04.605743 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.605575    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45794eea-ae6e-4638-8ea4-db648638cf41-kserve-provision-location\") pod \"45794eea-ae6e-4638-8ea4-db648638cf41\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") "
Jul 01 14:51:04.605743 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.605611    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-cabundle-cert\") pod \"45794eea-ae6e-4638-8ea4-db648638cf41\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") "
Jul 01 14:51:04.605743 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.605650    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-kube-rbac-proxy-sar-config\") pod \"45794eea-ae6e-4638-8ea4-db648638cf41\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") "
Jul 01 14:51:04.605874 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.605792    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45794eea-ae6e-4638-8ea4-db648638cf41-proxy-tls\") pod \"45794eea-ae6e-4638-8ea4-db648638cf41\" (UID: \"45794eea-ae6e-4638-8ea4-db648638cf41\") "
Jul 01 14:51:04.606009 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.605978    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45794eea-ae6e-4638-8ea4-db648638cf41-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45794eea-ae6e-4638-8ea4-db648638cf41" (UID: "45794eea-ae6e-4638-8ea4-db648638cf41"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:51:04.606009 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.605987    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45794eea-ae6e-4638-8ea4-db648638cf41" (UID: "45794eea-ae6e-4638-8ea4-db648638cf41"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:51:04.606168 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.606051    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45794eea-ae6e-4638-8ea4-db648638cf41" (UID: "45794eea-ae6e-4638-8ea4-db648638cf41"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:51:04.606168 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.606092    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45794eea-ae6e-4638-8ea4-db648638cf41-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:51:04.606168 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.606105    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:51:04.607746 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.607721    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45794eea-ae6e-4638-8ea4-db648638cf41-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45794eea-ae6e-4638-8ea4-db648638cf41" (UID: "45794eea-ae6e-4638-8ea4-db648638cf41"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:51:04.607819 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.607797    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45794eea-ae6e-4638-8ea4-db648638cf41-kube-api-access-stxgj" (OuterVolumeSpecName: "kube-api-access-stxgj") pod "45794eea-ae6e-4638-8ea4-db648638cf41" (UID: "45794eea-ae6e-4638-8ea4-db648638cf41"). InnerVolumeSpecName "kube-api-access-stxgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:51:04.707370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.707265    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45794eea-ae6e-4638-8ea4-db648638cf41-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:51:04.707370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.707303    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stxgj\" (UniqueName: \"kubernetes.io/projected/45794eea-ae6e-4638-8ea4-db648638cf41-kube-api-access-stxgj\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:51:04.707370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:04.707313    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45794eea-ae6e-4638-8ea4-db648638cf41-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:51:05.366613 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.366590    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:51:05.414348 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.414320    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdjqv\" (UniqueName: \"kubernetes.io/projected/22c2dede-364e-423f-938e-94b314e7f815-kube-api-access-bdjqv\") pod \"22c2dede-364e-423f-938e-94b314e7f815\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") "
Jul 01 14:51:05.414491 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.414365    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c2dede-364e-423f-938e-94b314e7f815-kserve-provision-location\") pod \"22c2dede-364e-423f-938e-94b314e7f815\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") "
Jul 01 14:51:05.414491 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.414413    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-cabundle-cert\") pod \"22c2dede-364e-423f-938e-94b314e7f815\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") "
Jul 01 14:51:05.414491 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.414465    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-kube-rbac-proxy-sar-config\") pod \"22c2dede-364e-423f-938e-94b314e7f815\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") "
Jul 01 14:51:05.414651 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.414526    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c2dede-364e-423f-938e-94b314e7f815-proxy-tls\") pod \"22c2dede-364e-423f-938e-94b314e7f815\" (UID: \"22c2dede-364e-423f-938e-94b314e7f815\") "
Jul 01 14:51:05.414786 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.414761    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c2dede-364e-423f-938e-94b314e7f815-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22c2dede-364e-423f-938e-94b314e7f815" (UID: "22c2dede-364e-423f-938e-94b314e7f815"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:51:05.414829 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.414769    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22c2dede-364e-423f-938e-94b314e7f815" (UID: "22c2dede-364e-423f-938e-94b314e7f815"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:51:05.414880 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.414831    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22c2dede-364e-423f-938e-94b314e7f815" (UID: "22c2dede-364e-423f-938e-94b314e7f815"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:51:05.416392 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.416372    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c2dede-364e-423f-938e-94b314e7f815-kube-api-access-bdjqv" (OuterVolumeSpecName: "kube-api-access-bdjqv") pod "22c2dede-364e-423f-938e-94b314e7f815" (UID: "22c2dede-364e-423f-938e-94b314e7f815"). InnerVolumeSpecName "kube-api-access-bdjqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:51:05.416507 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.416491    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c2dede-364e-423f-938e-94b314e7f815-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22c2dede-364e-423f-938e-94b314e7f815" (UID: "22c2dede-364e-423f-938e-94b314e7f815"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:51:05.435397 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.435373    2558 generic.go:356] "Generic (PLEG): container finished" podID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerID="05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3" exitCode=0
Jul 01 14:51:05.435767 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.435445    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" event={"ID":"b056f5a5-627b-4263-8487-0bff33f86e6e","Type":"ContainerDied","Data":"05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3"}
Jul 01 14:51:05.437471 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.437455    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"
Jul 01 14:51:05.437560 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.437477    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9" event={"ID":"45794eea-ae6e-4638-8ea4-db648638cf41","Type":"ContainerDied","Data":"5c53304c985ff1bf0d0295f4007c1b7517ba536fa68d03605d01ce6b34ca6091"}
Jul 01 14:51:05.437560 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.437550    2558 scope.go:117] "RemoveContainer" containerID="e94f914c60c959d569b2d7f48ca979396041c48192b70c8d8e2ba33e253d974d"
Jul 01 14:51:05.439610 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.439550    2558 generic.go:356] "Generic (PLEG): container finished" podID="22c2dede-364e-423f-938e-94b314e7f815" containerID="f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92" exitCode=0
Jul 01 14:51:05.439714 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.439620    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" event={"ID":"22c2dede-364e-423f-938e-94b314e7f815","Type":"ContainerDied","Data":"f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92"}
Jul 01 14:51:05.439789 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.439678    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6" event={"ID":"22c2dede-364e-423f-938e-94b314e7f815","Type":"ContainerDied","Data":"8aaeee3e812e10c4a47eedfe4e2a990650daa4b664bc30d5af7f4924403d0327"}
Jul 01 14:51:05.439789 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.439626    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"
Jul 01 14:51:05.441557 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.441540    2558 generic.go:356] "Generic (PLEG): container finished" podID="d788da9c-cd56-43aa-837b-aee97423de11" containerID="917b2a4ab62d714485fdb224f109cd1edd7d29dba3e41e5dc36d76b001f3af57" exitCode=0
Jul 01 14:51:05.441665 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.441590    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" event={"ID":"d788da9c-cd56-43aa-837b-aee97423de11","Type":"ContainerDied","Data":"917b2a4ab62d714485fdb224f109cd1edd7d29dba3e41e5dc36d76b001f3af57"}
Jul 01 14:51:05.453223 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.453203    2558 scope.go:117] "RemoveContainer" containerID="ef12ee51db9f0d6dacc043b19106d79ec4c799e12cface4725e4f604c7aeb81b"
Jul 01 14:51:05.464665 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.464647    2558 scope.go:117] "RemoveContainer" containerID="f117d8094b457788167b2bfd7c65c0dcd4e116f49083865e284e17378aff6d08"
Jul 01 14:51:05.472878 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.472856    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"]
Jul 01 14:51:05.477380 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.477360    2558 scope.go:117] "RemoveContainer" containerID="737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0"
Jul 01 14:51:05.480812 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.480788    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-63a63-predictor-8b657594b-jsqm6"]
Jul 01 14:51:05.487156 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.487137    2558 scope.go:117] "RemoveContainer" containerID="f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92"
Jul 01 14:51:05.494131 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.494105    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"]
Jul 01 14:51:05.500122 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.500098    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-63a63-predictor-5866499bcc-xtcv9"]
Jul 01 14:51:05.503928 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.503912    2558 scope.go:117] "RemoveContainer" containerID="d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323"
Jul 01 14:51:05.512355 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.512327    2558 scope.go:117] "RemoveContainer" containerID="737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0"
Jul 01 14:51:05.512600 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:51:05.512584    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0\": container with ID starting with 737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0 not found: ID does not exist" containerID="737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0"
Jul 01 14:51:05.512657 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.512607    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0"} err="failed to get container status \"737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0\": rpc error: code = NotFound desc = could not find container \"737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0\": container with ID starting with 737873659ab9e97b71a4cdabc46ce138e4328ea74a5b1b156f7612b89ce9ebf0 not found: ID does not exist"
Jul 01 14:51:05.512657 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.512623    2558 scope.go:117] "RemoveContainer" containerID="f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92"
Jul 01 14:51:05.512966 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:51:05.512947    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92\": container with ID starting with f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92 not found: ID does not exist" containerID="f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92"
Jul 01 14:51:05.513034 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.512972    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92"} err="failed to get container status \"f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92\": rpc error: code = NotFound desc = could not find container \"f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92\": container with ID starting with f05de2a3a853a1357df9e14294ea9d448cbb88df95cbab83e500e20323144e92 not found: ID does not exist"
Jul 01 14:51:05.513034 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.512988    2558 scope.go:117] "RemoveContainer" containerID="d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323"
Jul 01 14:51:05.513242 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:51:05.513227    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323\": container with ID starting with d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323 not found: ID does not exist" containerID="d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323"
Jul 01 14:51:05.513277 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.513247    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323"} err="failed to get container status \"d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323\": rpc error: code = NotFound desc = could not find container \"d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323\": container with ID starting with d63be6060eae48b10f94de6eb488231362167212b83ed19baff85f90cf432323 not found: ID does not exist"
Jul 01 14:51:05.516438 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.516415    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdjqv\" (UniqueName: \"kubernetes.io/projected/22c2dede-364e-423f-938e-94b314e7f815-kube-api-access-bdjqv\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:51:05.516539 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.516441    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c2dede-364e-423f-938e-94b314e7f815-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:51:05.516539 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.516485    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:51:05.516539 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.516498    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22c2dede-364e-423f-938e-94b314e7f815-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:51:05.516539 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:05.516512    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c2dede-364e-423f-938e-94b314e7f815-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:51:06.353554 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.353521    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c2dede-364e-423f-938e-94b314e7f815" path="/var/lib/kubelet/pods/22c2dede-364e-423f-938e-94b314e7f815/volumes"
Jul 01 14:51:06.354035 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.354021    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" path="/var/lib/kubelet/pods/45794eea-ae6e-4638-8ea4-db648638cf41/volumes"
Jul 01 14:51:06.448470 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.448437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" event={"ID":"b056f5a5-627b-4263-8487-0bff33f86e6e","Type":"ContainerStarted","Data":"3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e"}
Jul 01 14:51:06.448891 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.448496    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" event={"ID":"b056f5a5-627b-4263-8487-0bff33f86e6e","Type":"ContainerStarted","Data":"0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12"}
Jul 01 14:51:06.448891 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.448825    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:06.448986 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.448947    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:06.450448 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.450423    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 14:51:06.452263 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.452244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" event={"ID":"d788da9c-cd56-43aa-837b-aee97423de11","Type":"ContainerStarted","Data":"3ec0deb3962e165c784de30fc458e95f9ca7c1424926555d565e0fc6c798e87f"}
Jul 01 14:51:06.452358 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.452291    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" event={"ID":"d788da9c-cd56-43aa-837b-aee97423de11","Type":"ContainerStarted","Data":"13598fefcd2af1e31a5c0ca40e2cb601a63c024b2845c495083d18b3240c4fe7"}
Jul 01 14:51:06.452618 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.452596    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:06.452713 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.452651    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:06.453671 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.453652    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 14:51:06.475669 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.475631    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podStartSLOduration=2.890849203 podStartE2EDuration="6.475618035s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.584768832s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:51:06.475158107 +0000 UTC m=+714.535737298" watchObservedRunningTime="2026-07-01 14:51:06.475618035 +0000 UTC m=+714.536197225"
Jul 01 14:51:06.497113 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:06.497068    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podStartSLOduration=2.869893017 podStartE2EDuration="6.497052385s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.627159368s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:51:06.496797809 +0000 UTC m=+714.557377000" watchObservedRunningTime="2026-07-01 14:51:06.497052385 +0000 UTC m=+714.557631577"
Jul 01 14:51:07.459244 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:07.459206    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 14:51:07.459701 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:07.459297    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 14:51:12.465037 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:12.465006    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:51:12.465508 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:12.465489    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:51:12.465555 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:12.465524    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 14:51:12.465954 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:12.465927    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 14:51:22.466052 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:22.466006    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 14:51:22.466052 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:22.466026    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 14:51:32.466211 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:32.466173    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 14:51:32.466752 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:32.466188    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 14:51:42.466209 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:42.466168    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 14:51:42.466599 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:42.466168    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 14:51:52.466415 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:52.466377    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 14:51:52.466813 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:51:52.466377    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 14:52:02.465871 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:02.465835    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 14:52:02.466240 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:02.465948    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 14:52:12.465828 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:12.465800    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:52:12.466388 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:12.466051    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 14:52:22.467102 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:22.467074    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:52:51.023316 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023183    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"]
Jul 01 14:52:51.023666 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023633    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kube-rbac-proxy"
Jul 01 14:52:51.023666 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023647    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kube-rbac-proxy"
Jul 01 14:52:51.023666 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023661    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container"
Jul 01 14:52:51.023666 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023666    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023676    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="storage-initializer"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023682    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="storage-initializer"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023710    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023715    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023723    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kube-rbac-proxy"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023729    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kube-rbac-proxy"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023744    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="storage-initializer"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023749    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="storage-initializer"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023806    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kube-rbac-proxy"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023816    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kserve-container"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023821    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="45794eea-ae6e-4638-8ea4-db648638cf41" containerName="kube-rbac-proxy"
Jul 01 14:52:51.023823 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.023827    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c2dede-364e-423f-938e-94b314e7f815" containerName="kserve-container"
Jul 01 14:52:51.026831 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.026815    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.029457 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.029434    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4ce12-predictor-serving-cert\""
Jul 01 14:52:51.029665 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.029654    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4ce12-kube-rbac-proxy-sar-config\""
Jul 01 14:52:51.041911 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.041892    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"]
Jul 01 14:52:51.121227 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.121197    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") " pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.121399 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.121233    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8x6\" (UniqueName: \"kubernetes.io/projected/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-api-access-gr8x6\") pod \"message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") " pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.121399 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.121270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-proxy-tls\") pod \"message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") " pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.127548 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.127520    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"]
Jul 01 14:52:51.127821 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.127792    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" containerID="cri-o://0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12" gracePeriod=30
Jul 01 14:52:51.127922 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.127820    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kube-rbac-proxy" containerID="cri-o://3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e" gracePeriod=30
Jul 01 14:52:51.199366 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.199328    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"]
Jul 01 14:52:51.199620 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.199576    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container" containerID="cri-o://13598fefcd2af1e31a5c0ca40e2cb601a63c024b2845c495083d18b3240c4fe7" gracePeriod=30
Jul 01 14:52:51.199728 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.199602    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kube-rbac-proxy" containerID="cri-o://3ec0deb3962e165c784de30fc458e95f9ca7c1424926555d565e0fc6c798e87f" gracePeriod=30
Jul 01 14:52:51.222012 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.221988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") " pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.222184 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.222019    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8x6\" (UniqueName: \"kubernetes.io/projected/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-api-access-gr8x6\") pod \"message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") " pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.222184 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.222044    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-proxy-tls\") pod \"message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") " pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.222813 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.222782    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") " pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.224584 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.224564    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-proxy-tls\") pod \"message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") " pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.235787 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.235755    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8x6\" (UniqueName: \"kubernetes.io/projected/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-api-access-gr8x6\") pod \"message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") " pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.336521 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.336495    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:51.471226 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.471200    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"]
Jul 01 14:52:51.473639 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:52:51.473616    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbc12fa_60d5_4fe9_83f3_bb345d08bc00.slice/crio-f6d80b7214d5294fd02c4386f37c39dd7f1eef5d89defacae3120be33c85d73d WatchSource:0}: Error finding container f6d80b7214d5294fd02c4386f37c39dd7f1eef5d89defacae3120be33c85d73d: Status 404 returned error can't find the container with id f6d80b7214d5294fd02c4386f37c39dd7f1eef5d89defacae3120be33c85d73d
Jul 01 14:52:51.475445 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.475431    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 14:52:51.892829 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.892794    2558 generic.go:356] "Generic (PLEG): container finished" podID="d788da9c-cd56-43aa-837b-aee97423de11" containerID="3ec0deb3962e165c784de30fc458e95f9ca7c1424926555d565e0fc6c798e87f" exitCode=2
Jul 01 14:52:51.893032 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.892868    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" event={"ID":"d788da9c-cd56-43aa-837b-aee97423de11","Type":"ContainerDied","Data":"3ec0deb3962e165c784de30fc458e95f9ca7c1424926555d565e0fc6c798e87f"}
Jul 01 14:52:51.895058 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.895036    2558 generic.go:356] "Generic (PLEG): container finished" podID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerID="3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e" exitCode=2
Jul 01 14:52:51.895170 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.895113    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" event={"ID":"b056f5a5-627b-4263-8487-0bff33f86e6e","Type":"ContainerDied","Data":"3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e"}
Jul 01 14:52:51.896344 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:51.896318    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z" event={"ID":"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00","Type":"ContainerStarted","Data":"f6d80b7214d5294fd02c4386f37c39dd7f1eef5d89defacae3120be33c85d73d"}
Jul 01 14:52:52.459778 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:52.459742    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 01 14:52:52.460202 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:52.459741    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 01 14:52:52.466250 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:52.466217    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 14:52:52.466250 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:52.466234    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 14:52:53.906049 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:53.906014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z" event={"ID":"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00","Type":"ContainerStarted","Data":"04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd"}
Jul 01 14:52:53.906499 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:53.906072    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z" event={"ID":"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00","Type":"ContainerStarted","Data":"3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc"}
Jul 01 14:52:53.906499 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:53.906098    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:53.906499 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:53.906151    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:53.907821 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:53.907803    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:52:53.927284 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:53.927239    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z" podStartSLOduration=1.4684209799999999 podStartE2EDuration="2.927226972s" totalImagesPullingTime="1.458805992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:52:53.926515121 +0000 UTC m=+821.987094312" watchObservedRunningTime="2026-07-01 14:52:53.927226972 +0000 UTC m=+821.987806161"
Jul 01 14:52:54.914559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:54.914526    2558 generic.go:356] "Generic (PLEG): container finished" podID="d788da9c-cd56-43aa-837b-aee97423de11" containerID="13598fefcd2af1e31a5c0ca40e2cb601a63c024b2845c495083d18b3240c4fe7" exitCode=0
Jul 01 14:52:54.914897 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:54.914576    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" event={"ID":"d788da9c-cd56-43aa-837b-aee97423de11","Type":"ContainerDied","Data":"13598fefcd2af1e31a5c0ca40e2cb601a63c024b2845c495083d18b3240c4fe7"}
Jul 01 14:52:54.940861 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:54.940840    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:52:55.054710 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.054610    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgsv6\" (UniqueName: \"kubernetes.io/projected/d788da9c-cd56-43aa-837b-aee97423de11-kube-api-access-pgsv6\") pod \"d788da9c-cd56-43aa-837b-aee97423de11\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") "
Jul 01 14:52:55.054710 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.054649    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d788da9c-cd56-43aa-837b-aee97423de11-kserve-provision-location\") pod \"d788da9c-cd56-43aa-837b-aee97423de11\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") "
Jul 01 14:52:55.054710 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.054674    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-cabundle-cert\") pod \"d788da9c-cd56-43aa-837b-aee97423de11\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") "
Jul 01 14:52:55.054987 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.054718    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-kube-rbac-proxy-sar-config\") pod \"d788da9c-cd56-43aa-837b-aee97423de11\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") "
Jul 01 14:52:55.054987 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.054771    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d788da9c-cd56-43aa-837b-aee97423de11-proxy-tls\") pod \"d788da9c-cd56-43aa-837b-aee97423de11\" (UID: \"d788da9c-cd56-43aa-837b-aee97423de11\") "
Jul 01 14:52:55.055091 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.055021    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d788da9c-cd56-43aa-837b-aee97423de11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d788da9c-cd56-43aa-837b-aee97423de11" (UID: "d788da9c-cd56-43aa-837b-aee97423de11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:52:55.055091 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.055028    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d788da9c-cd56-43aa-837b-aee97423de11" (UID: "d788da9c-cd56-43aa-837b-aee97423de11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:52:55.055196 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.055088    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d788da9c-cd56-43aa-837b-aee97423de11" (UID: "d788da9c-cd56-43aa-837b-aee97423de11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:52:55.055196 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.055125    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d788da9c-cd56-43aa-837b-aee97423de11-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:52:55.055196 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.055144    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:52:55.056801 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.056782    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d788da9c-cd56-43aa-837b-aee97423de11-kube-api-access-pgsv6" (OuterVolumeSpecName: "kube-api-access-pgsv6") pod "d788da9c-cd56-43aa-837b-aee97423de11" (UID: "d788da9c-cd56-43aa-837b-aee97423de11"). InnerVolumeSpecName "kube-api-access-pgsv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:52:55.056901 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.056850    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d788da9c-cd56-43aa-837b-aee97423de11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d788da9c-cd56-43aa-837b-aee97423de11" (UID: "d788da9c-cd56-43aa-837b-aee97423de11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:52:55.156461 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.156416    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgsv6\" (UniqueName: \"kubernetes.io/projected/d788da9c-cd56-43aa-837b-aee97423de11-kube-api-access-pgsv6\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:52:55.156461 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.156456    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d788da9c-cd56-43aa-837b-aee97423de11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:52:55.156461 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.156466    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d788da9c-cd56-43aa-837b-aee97423de11-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:52:55.754503 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.754478    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:52:55.862365 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.862304    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b056f5a5-627b-4263-8487-0bff33f86e6e-proxy-tls\") pod \"b056f5a5-627b-4263-8487-0bff33f86e6e\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") "
Jul 01 14:52:55.862365 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.862357    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-rbac-proxy-sar-config\") pod \"b056f5a5-627b-4263-8487-0bff33f86e6e\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") "
Jul 01 14:52:55.862529 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.862393    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-cabundle-cert\") pod \"b056f5a5-627b-4263-8487-0bff33f86e6e\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") "
Jul 01 14:52:55.862529 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.862424    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltnqz\" (UniqueName: \"kubernetes.io/projected/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-api-access-ltnqz\") pod \"b056f5a5-627b-4263-8487-0bff33f86e6e\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") "
Jul 01 14:52:55.862529 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.862451    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b056f5a5-627b-4263-8487-0bff33f86e6e-kserve-provision-location\") pod \"b056f5a5-627b-4263-8487-0bff33f86e6e\" (UID: \"b056f5a5-627b-4263-8487-0bff33f86e6e\") "
Jul 01 14:52:55.862817 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.862796    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b056f5a5-627b-4263-8487-0bff33f86e6e" (UID: "b056f5a5-627b-4263-8487-0bff33f86e6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:52:55.862899 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.862801    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b056f5a5-627b-4263-8487-0bff33f86e6e" (UID: "b056f5a5-627b-4263-8487-0bff33f86e6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:52:55.862899 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.862844    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b056f5a5-627b-4263-8487-0bff33f86e6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b056f5a5-627b-4263-8487-0bff33f86e6e" (UID: "b056f5a5-627b-4263-8487-0bff33f86e6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:52:55.864407 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.864387    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-api-access-ltnqz" (OuterVolumeSpecName: "kube-api-access-ltnqz") pod "b056f5a5-627b-4263-8487-0bff33f86e6e" (UID: "b056f5a5-627b-4263-8487-0bff33f86e6e"). InnerVolumeSpecName "kube-api-access-ltnqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:52:55.864500 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.864446    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b056f5a5-627b-4263-8487-0bff33f86e6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b056f5a5-627b-4263-8487-0bff33f86e6e" (UID: "b056f5a5-627b-4263-8487-0bff33f86e6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:52:55.920529 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.920504    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"
Jul 01 14:52:55.920529 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.920520    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6" event={"ID":"d788da9c-cd56-43aa-837b-aee97423de11","Type":"ContainerDied","Data":"e41806bccc31d6e0cf815f2da08f2e1d5d5345444ff0c46e35bae75bc0920f62"}
Jul 01 14:52:55.920990 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.920587    2558 scope.go:117] "RemoveContainer" containerID="3ec0deb3962e165c784de30fc458e95f9ca7c1424926555d565e0fc6c798e87f"
Jul 01 14:52:55.922518 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.922493    2558 generic.go:356] "Generic (PLEG): container finished" podID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerID="0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12" exitCode=0
Jul 01 14:52:55.922624 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.922562    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"
Jul 01 14:52:55.922624 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.922576    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" event={"ID":"b056f5a5-627b-4263-8487-0bff33f86e6e","Type":"ContainerDied","Data":"0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12"}
Jul 01 14:52:55.922764 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.922637    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4" event={"ID":"b056f5a5-627b-4263-8487-0bff33f86e6e","Type":"ContainerDied","Data":"cccf634ec20822620caf1192eff7730a3333bcfe05f3568a3f611d256a081da8"}
Jul 01 14:52:55.929335 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.929313    2558 scope.go:117] "RemoveContainer" containerID="13598fefcd2af1e31a5c0ca40e2cb601a63c024b2845c495083d18b3240c4fe7"
Jul 01 14:52:55.936774 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.936754    2558 scope.go:117] "RemoveContainer" containerID="917b2a4ab62d714485fdb224f109cd1edd7d29dba3e41e5dc36d76b001f3af57"
Jul 01 14:52:55.946239 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.946210    2558 scope.go:117] "RemoveContainer" containerID="3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e"
Jul 01 14:52:55.954043 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.954020    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"]
Jul 01 14:52:55.954679 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.954665    2558 scope.go:117] "RemoveContainer" containerID="0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12"
Jul 01 14:52:55.959732 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.959713    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-02737-predictor-5c69dc797c-s77m6"]
Jul 01 14:52:55.962286 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.962265    2558 scope.go:117] "RemoveContainer" containerID="05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3"
Jul 01 14:52:55.963033 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.963017    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:52:55.963096 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.963039    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltnqz\" (UniqueName: \"kubernetes.io/projected/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-api-access-ltnqz\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:52:55.963096 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.963056    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b056f5a5-627b-4263-8487-0bff33f86e6e-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:52:55.963096 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.963066    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b056f5a5-627b-4263-8487-0bff33f86e6e-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:52:55.963096 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.963075    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b056f5a5-627b-4263-8487-0bff33f86e6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:52:55.969187 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.969169    2558 scope.go:117] "RemoveContainer" containerID="3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e"
Jul 01 14:52:55.969428 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:52:55.969406    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e\": container with ID starting with 3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e not found: ID does not exist" containerID="3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e"
Jul 01 14:52:55.969496 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.969440    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e"} err="failed to get container status \"3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e\": rpc error: code = NotFound desc = could not find container \"3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e\": container with ID starting with 3dee4c07a1b1a4fec42b71541abb34ca4be919680b0347637bb701248a30f91e not found: ID does not exist"
Jul 01 14:52:55.969496 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.969462    2558 scope.go:117] "RemoveContainer" containerID="0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12"
Jul 01 14:52:55.969729 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:52:55.969705    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12\": container with ID starting with 0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12 not found: ID does not exist" containerID="0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12"
Jul 01 14:52:55.969836 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.969738    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12"} err="failed to get container status \"0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12\": rpc error: code = NotFound desc = could not find container \"0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12\": container with ID starting with 0ec85453b733cdfb5b7a07824de0227919012cf79a294c885512e0d0720adb12 not found: ID does not exist"
Jul 01 14:52:55.969836 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.969760    2558 scope.go:117] "RemoveContainer" containerID="05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3"
Jul 01 14:52:55.970031 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:52:55.970013    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3\": container with ID starting with 05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3 not found: ID does not exist" containerID="05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3"
Jul 01 14:52:55.970071 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.970038    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3"} err="failed to get container status \"05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3\": rpc error: code = NotFound desc = could not find container \"05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3\": container with ID starting with 05ff287555c47a9c81f0f21d4043c409bc8261aa510012d018378999aebf98e3 not found: ID does not exist"
Jul 01 14:52:55.978097 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.978076    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"]
Jul 01 14:52:55.988496 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:55.988478    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-02737-predictor-b7bc4b6b6-x5bg4"]
Jul 01 14:52:56.353340 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:56.353310    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" path="/var/lib/kubelet/pods/b056f5a5-627b-4263-8487-0bff33f86e6e/volumes"
Jul 01 14:52:56.353811 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:52:56.353798    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d788da9c-cd56-43aa-837b-aee97423de11" path="/var/lib/kubelet/pods/d788da9c-cd56-43aa-837b-aee97423de11/volumes"
Jul 01 14:53:00.927123 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:00.927089    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:53:10.988497 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988468    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"]
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988819    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988830    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988840    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kube-rbac-proxy"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988846    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kube-rbac-proxy"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988855    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="storage-initializer"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988861    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="storage-initializer"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988870    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988875    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988889    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="storage-initializer"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988893    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="storage-initializer"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988902    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kube-rbac-proxy"
Jul 01 14:53:10.988910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988907    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kube-rbac-proxy"
Jul 01 14:53:10.989245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988956    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kube-rbac-proxy"
Jul 01 14:53:10.989245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988966    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kserve-container"
Jul 01 14:53:10.989245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988973    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d788da9c-cd56-43aa-837b-aee97423de11" containerName="kube-rbac-proxy"
Jul 01 14:53:10.989245 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.988979    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b056f5a5-627b-4263-8487-0bff33f86e6e" containerName="kserve-container"
Jul 01 14:53:10.992237 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.992222    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:10.994757 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.994728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 14:53:10.994757 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.994679    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4ce12-predictor-serving-cert\""
Jul 01 14:53:10.994937 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.994812    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 14:53:10.994937 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:10.994728    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4ce12-kube-rbac-proxy-sar-config\""
Jul 01 14:53:11.007495 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.007467    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"]
Jul 01 14:53:11.097154 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.097114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.097342 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.097165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kserve-provision-location\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.097342 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.097242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b680726-af0f-4165-9c8b-0f59dd0d42a6-proxy-tls\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.097342 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.097318    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-cabundle-cert\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.097474 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.097365    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pp9l\" (UniqueName: \"kubernetes.io/projected/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-api-access-2pp9l\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.198252 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.198218    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.198419 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.198261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kserve-provision-location\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.198419 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.198301    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b680726-af0f-4165-9c8b-0f59dd0d42a6-proxy-tls\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.198419 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.198332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-cabundle-cert\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.198419 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.198363    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pp9l\" (UniqueName: \"kubernetes.io/projected/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-api-access-2pp9l\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.198791 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.198768    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kserve-provision-location\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.199019 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.198998    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.199118 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.199098    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-cabundle-cert\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.200771 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.200750    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b680726-af0f-4165-9c8b-0f59dd0d42a6-proxy-tls\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.210554 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.210524    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pp9l\" (UniqueName: \"kubernetes.io/projected/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-api-access-2pp9l\") pod \"isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.303028 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.302932    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:11.448952 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.448923    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"]
Jul 01 14:53:11.449106 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:53:11.449081    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b680726_af0f_4165_9c8b_0f59dd0d42a6.slice/crio-88db935a82d34ae24b1b328badd4371f0919a5e985bbab1832d78d6fdcc3faf3 WatchSource:0}: Error finding container 88db935a82d34ae24b1b328badd4371f0919a5e985bbab1832d78d6fdcc3faf3: Status 404 returned error can't find the container with id 88db935a82d34ae24b1b328badd4371f0919a5e985bbab1832d78d6fdcc3faf3
Jul 01 14:53:11.995748 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.995714    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" event={"ID":"5b680726-af0f-4165-9c8b-0f59dd0d42a6","Type":"ContainerStarted","Data":"5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3"}
Jul 01 14:53:11.996234 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:11.995791    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" event={"ID":"5b680726-af0f-4165-9c8b-0f59dd0d42a6","Type":"ContainerStarted","Data":"88db935a82d34ae24b1b328badd4371f0919a5e985bbab1832d78d6fdcc3faf3"}
Jul 01 14:53:16.013512 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:16.013478    2558 generic.go:356] "Generic (PLEG): container finished" podID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerID="5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3" exitCode=0
Jul 01 14:53:16.013893 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:16.013549    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" event={"ID":"5b680726-af0f-4165-9c8b-0f59dd0d42a6","Type":"ContainerDied","Data":"5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3"}
Jul 01 14:53:17.020665 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:17.020630    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" event={"ID":"5b680726-af0f-4165-9c8b-0f59dd0d42a6","Type":"ContainerStarted","Data":"6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3"}
Jul 01 14:53:17.021074 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:17.020712    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" event={"ID":"5b680726-af0f-4165-9c8b-0f59dd0d42a6","Type":"ContainerStarted","Data":"7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2"}
Jul 01 14:53:17.021074 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:17.020737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" event={"ID":"5b680726-af0f-4165-9c8b-0f59dd0d42a6","Type":"ContainerStarted","Data":"4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4"}
Jul 01 14:53:17.021074 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:17.021005    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:17.021272 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:17.021149    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:17.022206 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:17.022183    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:53:17.069834 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:17.069790    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podStartSLOduration=3.568755196 podStartE2EDuration="7.069777219s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.501022023s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:53:17.068959361 +0000 UTC m=+845.129538555" watchObservedRunningTime="2026-07-01 14:53:17.069777219 +0000 UTC m=+845.130356410"
Jul 01 14:53:18.025711 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:18.025660    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:18.026156 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:18.025807    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:53:18.026872 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:18.026849    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:19.029586 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:19.029542    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:53:19.029998 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:19.029940    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:24.034747 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:24.034716    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:53:24.035406 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:24.035380    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:53:24.035569 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:24.035536    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:34.035594 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:34.035549    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:53:34.036071 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:34.035960    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:44.035299 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:44.035261    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:53:44.035805 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:44.035610    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:54.035548 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:54.035503    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:53:54.036063 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:53:54.035931    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:54:04.035365 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:04.035325    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:54:04.035851 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:04.035766    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:54:12.370313 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:12.370239    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:54:12.370836 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:12.370330    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:54:14.036200 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:14.036152    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:54:14.036648 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:14.036624    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:54:24.036148 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:24.036105    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:54:24.036622 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:24.036523    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:54:34.036286 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:34.036249    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:54:34.038749 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:34.036374    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:54:45.977186 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:45.977160    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z_8fbc12fa-60d5-4fe9-83f3-bb345d08bc00/kserve-container/0.log"
Jul 01 14:54:46.304431 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.304352    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"]
Jul 01 14:54:46.304766 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.304717    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" containerID="cri-o://4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4" gracePeriod=30
Jul 01 14:54:46.304872 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.304749    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" containerID="cri-o://6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3" gracePeriod=30
Jul 01 14:54:46.304872 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.304792    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kube-rbac-proxy" containerID="cri-o://7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2" gracePeriod=30
Jul 01 14:54:46.314924 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.314903    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"]
Jul 01 14:54:46.318254 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.318239    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.320598 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.320582    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-b7013-predictor-serving-cert\""
Jul 01 14:54:46.320671 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.320605    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-b7013-kube-rbac-proxy-sar-config\""
Jul 01 14:54:46.329160 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.329140    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"]
Jul 01 14:54:46.365561 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.365537    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"]
Jul 01 14:54:46.365832 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.365805    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z" podUID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" containerName="kube-rbac-proxy" containerID="cri-o://04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd" gracePeriod=30
Jul 01 14:54:46.365957 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.365793    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z" podUID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" containerName="kserve-container" containerID="cri-o://3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc" gracePeriod=30
Jul 01 14:54:46.391777 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.391752    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.391884 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.391811    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807a9b97-34bf-4546-8f35-5d378f8806d9-proxy-tls\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.391884 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.391868    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kzcx\" (UniqueName: \"kubernetes.io/projected/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-api-access-5kzcx\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.392013 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.391950    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/807a9b97-34bf-4546-8f35-5d378f8806d9-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.392013 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.391982    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.492473 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.492445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.492626 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.492521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807a9b97-34bf-4546-8f35-5d378f8806d9-proxy-tls\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.492626 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.492551    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kzcx\" (UniqueName: \"kubernetes.io/projected/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-api-access-5kzcx\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.492626 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.492614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/807a9b97-34bf-4546-8f35-5d378f8806d9-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.492820 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.492643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.492820 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:54:46.492801    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-serving-cert: secret "isvc-sklearn-scale-raw-b7013-predictor-serving-cert" not found
Jul 01 14:54:46.492924 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:54:46.492871    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/807a9b97-34bf-4546-8f35-5d378f8806d9-proxy-tls podName:807a9b97-34bf-4546-8f35-5d378f8806d9 nodeName:}" failed. No retries permitted until 2026-07-01 14:54:46.9928499 +0000 UTC m=+935.053429088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/807a9b97-34bf-4546-8f35-5d378f8806d9-proxy-tls") pod "isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" (UID: "807a9b97-34bf-4546-8f35-5d378f8806d9") : secret "isvc-sklearn-scale-raw-b7013-predictor-serving-cert" not found
Jul 01 14:54:46.493042 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.493023    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/807a9b97-34bf-4546-8f35-5d378f8806d9-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.493225 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.493203    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.493485 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.493467    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.502447 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.502412    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kzcx\" (UniqueName: \"kubernetes.io/projected/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-api-access-5kzcx\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.607977 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.607956    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:54:46.693699 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.693664    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr8x6\" (UniqueName: \"kubernetes.io/projected/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-api-access-gr8x6\") pod \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") "
Jul 01 14:54:46.693852 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.693778    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-proxy-tls\") pod \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") "
Jul 01 14:54:46.693852 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.693806    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-rbac-proxy-sar-config\") pod \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\" (UID: \"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00\") "
Jul 01 14:54:46.694236 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.694207    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" (UID: "8fbc12fa-60d5-4fe9-83f3-bb345d08bc00"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:54:46.695772 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.695749    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" (UID: "8fbc12fa-60d5-4fe9-83f3-bb345d08bc00"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:54:46.695859 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.695807    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-api-access-gr8x6" (OuterVolumeSpecName: "kube-api-access-gr8x6") pod "8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" (UID: "8fbc12fa-60d5-4fe9-83f3-bb345d08bc00"). InnerVolumeSpecName "kube-api-access-gr8x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:54:46.795370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.795344    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:54:46.795370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.795368    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:54:46.795513 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.795378    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr8x6\" (UniqueName: \"kubernetes.io/projected/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00-kube-api-access-gr8x6\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:54:46.997627 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.997548    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807a9b97-34bf-4546-8f35-5d378f8806d9-proxy-tls\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:46.999830 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:46.999812    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807a9b97-34bf-4546-8f35-5d378f8806d9-proxy-tls\") pod \"isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:47.228842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.228806    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:47.362430 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.362391    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"]
Jul 01 14:54:47.365008 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:54:47.364981    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807a9b97_34bf_4546_8f35_5d378f8806d9.slice/crio-8e8f2f8995b741fd05e9f8d2d567ac64c51f9a8808c46627b97214577dae883f WatchSource:0}: Error finding container 8e8f2f8995b741fd05e9f8d2d567ac64c51f9a8808c46627b97214577dae883f: Status 404 returned error can't find the container with id 8e8f2f8995b741fd05e9f8d2d567ac64c51f9a8808c46627b97214577dae883f
Jul 01 14:54:47.411377 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.411351    2558 generic.go:356] "Generic (PLEG): container finished" podID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" containerID="04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd" exitCode=2
Jul 01 14:54:47.411377 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.411375    2558 generic.go:356] "Generic (PLEG): container finished" podID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" containerID="3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc" exitCode=2
Jul 01 14:54:47.411562 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.411385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z" event={"ID":"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00","Type":"ContainerDied","Data":"04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd"}
Jul 01 14:54:47.411562 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.411414    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"
Jul 01 14:54:47.411562 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.411453    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z" event={"ID":"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00","Type":"ContainerDied","Data":"3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc"}
Jul 01 14:54:47.411562 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.411470    2558 scope.go:117] "RemoveContainer" containerID="04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd"
Jul 01 14:54:47.411562 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.411486    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z" event={"ID":"8fbc12fa-60d5-4fe9-83f3-bb345d08bc00","Type":"ContainerDied","Data":"f6d80b7214d5294fd02c4386f37c39dd7f1eef5d89defacae3120be33c85d73d"}
Jul 01 14:54:47.414085 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.414066    2558 generic.go:356] "Generic (PLEG): container finished" podID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerID="7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2" exitCode=2
Jul 01 14:54:47.414201 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.414124    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" event={"ID":"5b680726-af0f-4165-9c8b-0f59dd0d42a6","Type":"ContainerDied","Data":"7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2"}
Jul 01 14:54:47.415350 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.415331    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" event={"ID":"807a9b97-34bf-4546-8f35-5d378f8806d9","Type":"ContainerStarted","Data":"8e8f2f8995b741fd05e9f8d2d567ac64c51f9a8808c46627b97214577dae883f"}
Jul 01 14:54:47.423024 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.423007    2558 scope.go:117] "RemoveContainer" containerID="3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc"
Jul 01 14:54:47.431037 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.431019    2558 scope.go:117] "RemoveContainer" containerID="04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd"
Jul 01 14:54:47.431302 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:54:47.431282    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd\": container with ID starting with 04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd not found: ID does not exist" containerID="04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd"
Jul 01 14:54:47.431353 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.431309    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd"} err="failed to get container status \"04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd\": rpc error: code = NotFound desc = could not find container \"04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd\": container with ID starting with 04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd not found: ID does not exist"
Jul 01 14:54:47.431353 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.431326    2558 scope.go:117] "RemoveContainer" containerID="3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc"
Jul 01 14:54:47.431622 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:54:47.431596    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc\": container with ID starting with 3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc not found: ID does not exist" containerID="3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc"
Jul 01 14:54:47.431785 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.431633    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc"} err="failed to get container status \"3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc\": rpc error: code = NotFound desc = could not find container \"3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc\": container with ID starting with 3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc not found: ID does not exist"
Jul 01 14:54:47.431785 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.431654    2558 scope.go:117] "RemoveContainer" containerID="04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd"
Jul 01 14:54:47.431949 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.431928    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd"} err="failed to get container status \"04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd\": rpc error: code = NotFound desc = could not find container \"04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd\": container with ID starting with 04d4fec15393654452cccfd43a0abebacb1c3f053daf008e83a9771e789da6cd not found: ID does not exist"
Jul 01 14:54:47.431994 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.431951    2558 scope.go:117] "RemoveContainer" containerID="3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc"
Jul 01 14:54:47.432196 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.432176    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc"} err="failed to get container status \"3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc\": rpc error: code = NotFound desc = could not find container \"3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc\": container with ID starting with 3daf6b1306a7b4355df609d2a7479e9e7dc7f1a9d0ecb256d7ce16759e6d43dc not found: ID does not exist"
Jul 01 14:54:47.436233 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.436209    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"]
Jul 01 14:54:47.440798 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:47.440779    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4ce12-predictor-9d84f76c9-4r95z"]
Jul 01 14:54:48.353382 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:48.353349    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" path="/var/lib/kubelet/pods/8fbc12fa-60d5-4fe9-83f3-bb345d08bc00/volumes"
Jul 01 14:54:48.427178 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:48.427145    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" event={"ID":"807a9b97-34bf-4546-8f35-5d378f8806d9","Type":"ContainerStarted","Data":"2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a"}
Jul 01 14:54:49.030277 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:49.030238    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 01 14:54:51.440919 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:51.440888    2558 generic.go:356] "Generic (PLEG): container finished" podID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerID="4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4" exitCode=0
Jul 01 14:54:51.441290 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:51.440954    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" event={"ID":"5b680726-af0f-4165-9c8b-0f59dd0d42a6","Type":"ContainerDied","Data":"4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4"}
Jul 01 14:54:51.442293 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:51.442272    2558 generic.go:356] "Generic (PLEG): container finished" podID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerID="2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a" exitCode=0
Jul 01 14:54:51.442377 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:51.442338    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" event={"ID":"807a9b97-34bf-4546-8f35-5d378f8806d9","Type":"ContainerDied","Data":"2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a"}
Jul 01 14:54:52.448879 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:52.448840    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" event={"ID":"807a9b97-34bf-4546-8f35-5d378f8806d9","Type":"ContainerStarted","Data":"93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20"}
Jul 01 14:54:52.449278 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:52.448895    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" event={"ID":"807a9b97-34bf-4546-8f35-5d378f8806d9","Type":"ContainerStarted","Data":"2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337"}
Jul 01 14:54:52.449278 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:52.449087    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:52.473344 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:52.473300    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podStartSLOduration=2.696689853 podStartE2EDuration="6.473286394s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.776596541s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:54:52.470647108 +0000 UTC m=+940.531226322" watchObservedRunningTime="2026-07-01 14:54:52.473286394 +0000 UTC m=+940.533865585"
Jul 01 14:54:53.453116 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:53.453081    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:53.454282 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:53.454258    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:54:54.030432 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:54.030387    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 01 14:54:54.035847 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:54.035810    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:54:54.036120 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:54.036101    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:54:54.457107 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:54.457070    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:54:59.029595 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:59.029551    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 01 14:54:59.029995 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:59.029701    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:54:59.462912 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:59.462885    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:54:59.463376 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:54:59.463352    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:55:04.030405 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:04.030369    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 01 14:55:04.035824 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:04.035800    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:55:04.036081 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:04.036052    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:55:09.030451 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:09.030407    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 01 14:55:09.464259 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:09.464220    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:55:14.030519 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:14.030476    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 01 14:55:14.035928 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:14.035892    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 14:55:14.036059 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:14.036035    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:55:14.036235 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:14.036212    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:55:14.036351 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:14.036337    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:55:16.457150 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.457123    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:55:16.545370 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.545336    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-cabundle-cert\") pod \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") "
Jul 01 14:55:16.545508 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.545382    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kserve-provision-location\") pod \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") "
Jul 01 14:55:16.545508 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.545405    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pp9l\" (UniqueName: \"kubernetes.io/projected/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-api-access-2pp9l\") pod \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") "
Jul 01 14:55:16.545600 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.545518    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-rbac-proxy-sar-config\") pod \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") "
Jul 01 14:55:16.545600 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.545570    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b680726-af0f-4165-9c8b-0f59dd0d42a6-proxy-tls\") pod \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\" (UID: \"5b680726-af0f-4165-9c8b-0f59dd0d42a6\") "
Jul 01 14:55:16.545733 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.545677    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b680726-af0f-4165-9c8b-0f59dd0d42a6" (UID: "5b680726-af0f-4165-9c8b-0f59dd0d42a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:55:16.545797 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.545737    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b680726-af0f-4165-9c8b-0f59dd0d42a6" (UID: "5b680726-af0f-4165-9c8b-0f59dd0d42a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:55:16.545927 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.545908    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b680726-af0f-4165-9c8b-0f59dd0d42a6" (UID: "5b680726-af0f-4165-9c8b-0f59dd0d42a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:55:16.546008 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.545992    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:55:16.546058 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.546010    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:55:16.546058 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.546024    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b680726-af0f-4165-9c8b-0f59dd0d42a6-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:55:16.547434 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.547406    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-api-access-2pp9l" (OuterVolumeSpecName: "kube-api-access-2pp9l") pod "5b680726-af0f-4165-9c8b-0f59dd0d42a6" (UID: "5b680726-af0f-4165-9c8b-0f59dd0d42a6"). InnerVolumeSpecName "kube-api-access-2pp9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:55:16.547555 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.547537    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b680726-af0f-4165-9c8b-0f59dd0d42a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b680726-af0f-4165-9c8b-0f59dd0d42a6" (UID: "5b680726-af0f-4165-9c8b-0f59dd0d42a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:55:16.559477 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.559423    2558 generic.go:356] "Generic (PLEG): container finished" podID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerID="6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3" exitCode=0
Jul 01 14:55:16.559562 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.559482    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"
Jul 01 14:55:16.559562 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.559504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" event={"ID":"5b680726-af0f-4165-9c8b-0f59dd0d42a6","Type":"ContainerDied","Data":"6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3"}
Jul 01 14:55:16.559631 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.559569    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8" event={"ID":"5b680726-af0f-4165-9c8b-0f59dd0d42a6","Type":"ContainerDied","Data":"88db935a82d34ae24b1b328badd4371f0919a5e985bbab1832d78d6fdcc3faf3"}
Jul 01 14:55:16.559631 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.559601    2558 scope.go:117] "RemoveContainer" containerID="6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3"
Jul 01 14:55:16.568388 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.568363    2558 scope.go:117] "RemoveContainer" containerID="7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2"
Jul 01 14:55:16.575407 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.575392    2558 scope.go:117] "RemoveContainer" containerID="4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4"
Jul 01 14:55:16.582089 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.582074    2558 scope.go:117] "RemoveContainer" containerID="5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3"
Jul 01 14:55:16.584963 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.584945    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"]
Jul 01 14:55:16.588992 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.588973    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4ce12-predictor-567557fdcb-fvjj8"]
Jul 01 14:55:16.589975 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.589961    2558 scope.go:117] "RemoveContainer" containerID="6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3"
Jul 01 14:55:16.590189 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:55:16.590169    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3\": container with ID starting with 6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3 not found: ID does not exist" containerID="6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3"
Jul 01 14:55:16.590233 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.590195    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3"} err="failed to get container status \"6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3\": rpc error: code = NotFound desc = could not find container \"6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3\": container with ID starting with 6e436948c99d9eb2ca4af5ac8eae8558d4ff3bfe1411cd74f598b8ffdbe86bc3 not found: ID does not exist"
Jul 01 14:55:16.590233 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.590214    2558 scope.go:117] "RemoveContainer" containerID="7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2"
Jul 01 14:55:16.590450 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:55:16.590434    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2\": container with ID starting with 7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2 not found: ID does not exist" containerID="7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2"
Jul 01 14:55:16.590499 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.590455    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2"} err="failed to get container status \"7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2\": rpc error: code = NotFound desc = could not find container \"7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2\": container with ID starting with 7426b7a3c138487b8a38ce6928e9d4dcb4131d614672b499723a5969d6d9f2b2 not found: ID does not exist"
Jul 01 14:55:16.590499 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.590469    2558 scope.go:117] "RemoveContainer" containerID="4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4"
Jul 01 14:55:16.590735 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:55:16.590717    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4\": container with ID starting with 4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4 not found: ID does not exist" containerID="4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4"
Jul 01 14:55:16.590780 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.590740    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4"} err="failed to get container status \"4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4\": rpc error: code = NotFound desc = could not find container \"4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4\": container with ID starting with 4f2e7aa6993da743583ec13f3d17333098b0a8b445fd2650ab3b8617911a3ef4 not found: ID does not exist"
Jul 01 14:55:16.590780 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.590756    2558 scope.go:117] "RemoveContainer" containerID="5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3"
Jul 01 14:55:16.590979 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:55:16.590964    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3\": container with ID starting with 5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3 not found: ID does not exist" containerID="5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3"
Jul 01 14:55:16.591021 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.590983    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3"} err="failed to get container status \"5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3\": rpc error: code = NotFound desc = could not find container \"5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3\": container with ID starting with 5d63b77c3f0b961a442c70b3b12363626f0eecafe5540998702e8a100addf0f3 not found: ID does not exist"
Jul 01 14:55:16.646507 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.646488    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pp9l\" (UniqueName: \"kubernetes.io/projected/5b680726-af0f-4165-9c8b-0f59dd0d42a6-kube-api-access-2pp9l\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:55:16.646507 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:16.646507    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b680726-af0f-4165-9c8b-0f59dd0d42a6-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:55:18.353438 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:18.353408    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" path="/var/lib/kubelet/pods/5b680726-af0f-4165-9c8b-0f59dd0d42a6/volumes"
Jul 01 14:55:19.463540 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:19.463506    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:55:29.463456 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:29.463414    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:55:39.463975 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:39.463888    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:55:49.463936 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:49.463891    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:55:59.463843 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:55:59.463673    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:56:02.351173 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:56:02.351126    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:56:12.351140 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:56:12.351099    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:56:22.351306 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:56:22.351265    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:56:32.351090 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:56:32.351053    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:56:42.351157 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:56:42.351105    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:56:52.351068 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:56:52.351023    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:57:02.351138 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:02.351097    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:57:12.351380 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:12.351301    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:57:22.352703 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:22.352667    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:57:26.480076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:26.480044    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"]
Jul 01 14:57:26.480589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:26.480290    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" containerID="cri-o://2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337" gracePeriod=30
Jul 01 14:57:26.480589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:26.480354    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kube-rbac-proxy" containerID="cri-o://93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20" gracePeriod=30
Jul 01 14:57:27.093063 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:27.093030    2558 generic.go:356] "Generic (PLEG): container finished" podID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerID="93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20" exitCode=2
Jul 01 14:57:27.093227 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:27.093082    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" event={"ID":"807a9b97-34bf-4546-8f35-5d378f8806d9","Type":"ContainerDied","Data":"93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20"}
Jul 01 14:57:29.457910 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:29.457869    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 14:57:31.557092 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557051    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"]
Jul 01 14:57:31.557591 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557572    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" containerName="kserve-container"
Jul 01 14:57:31.557670 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557594    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" containerName="kserve-container"
Jul 01 14:57:31.557670 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557607    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container"
Jul 01 14:57:31.557670 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557615    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container"
Jul 01 14:57:31.557670 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557629    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="storage-initializer"
Jul 01 14:57:31.557670 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557638    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="storage-initializer"
Jul 01 14:57:31.557670 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557649    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kube-rbac-proxy"
Jul 01 14:57:31.557670 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557657    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kube-rbac-proxy"
Jul 01 14:57:31.557670 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557670    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent"
Jul 01 14:57:31.558086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557679    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent"
Jul 01 14:57:31.558086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557705    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" containerName="kube-rbac-proxy"
Jul 01 14:57:31.558086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557713    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" containerName="kube-rbac-proxy"
Jul 01 14:57:31.558086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557822    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" containerName="kube-rbac-proxy"
Jul 01 14:57:31.558086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557834    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="agent"
Jul 01 14:57:31.558086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557844    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbc12fa-60d5-4fe9-83f3-bb345d08bc00" containerName="kserve-container"
Jul 01 14:57:31.558086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557854    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kube-rbac-proxy"
Jul 01 14:57:31.558086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.557864    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b680726-af0f-4165-9c8b-0f59dd0d42a6" containerName="kserve-container"
Jul 01 14:57:31.561309 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.561288    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.563656 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.563631    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-df069b-kube-rbac-proxy-sar-config\""
Jul 01 14:57:31.563775 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.563666    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-df069b-predictor-serving-cert\""
Jul 01 14:57:31.570890 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.570870    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"]
Jul 01 14:57:31.611854 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.611828    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a2cb72-1399-4e30-83aa-ccd8c293828d-proxy-tls\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.611973 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.611859    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-cabundle-cert\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.611973 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.611888    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.611973 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.611952    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kserve-provision-location\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.612075 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.612022    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7shdh\" (UniqueName: \"kubernetes.io/projected/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-api-access-7shdh\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.712761 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.712726    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-cabundle-cert\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.712906 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.712769    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.712906 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.712839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kserve-provision-location\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.712997 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.712909    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shdh\" (UniqueName: \"kubernetes.io/projected/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-api-access-7shdh\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.712997 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.712939    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a2cb72-1399-4e30-83aa-ccd8c293828d-proxy-tls\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.713309 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.713289    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kserve-provision-location\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.713513 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.713495    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.713559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.713497    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-cabundle-cert\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.715236 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.715220    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a2cb72-1399-4e30-83aa-ccd8c293828d-proxy-tls\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.722909 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.722881    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shdh\" (UniqueName: \"kubernetes.io/projected/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-api-access-7shdh\") pod \"isvc-primary-df069b-predictor-8597f6d579-wrvrh\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") " pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:31.871770 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:31.871742    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:32.016272 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:32.016235    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"]
Jul 01 14:57:32.019262 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:57:32.019235    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a2cb72_1399_4e30_83aa_ccd8c293828d.slice/crio-9dd73586c3668982ffa8bda745d2513cc245e73058180601459cdf3bb1792882 WatchSource:0}: Error finding container 9dd73586c3668982ffa8bda745d2513cc245e73058180601459cdf3bb1792882: Status 404 returned error can't find the container with id 9dd73586c3668982ffa8bda745d2513cc245e73058180601459cdf3bb1792882
Jul 01 14:57:32.115101 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:32.115065    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" event={"ID":"c5a2cb72-1399-4e30-83aa-ccd8c293828d","Type":"ContainerStarted","Data":"7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87"}
Jul 01 14:57:32.115270 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:32.115135    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" event={"ID":"c5a2cb72-1399-4e30-83aa-ccd8c293828d","Type":"ContainerStarted","Data":"9dd73586c3668982ffa8bda745d2513cc245e73058180601459cdf3bb1792882"}
Jul 01 14:57:32.351239 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:32.351199    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 14:57:34.457875 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:34.457831    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 14:57:36.132056 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.131971    2558 generic.go:356] "Generic (PLEG): container finished" podID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerID="7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87" exitCode=0
Jul 01 14:57:36.132056 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.132040    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" event={"ID":"c5a2cb72-1399-4e30-83aa-ccd8c293828d","Type":"ContainerDied","Data":"7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87"}
Jul 01 14:57:36.320470 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.320449    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:57:36.354238 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.354214    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kzcx\" (UniqueName: \"kubernetes.io/projected/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-api-access-5kzcx\") pod \"807a9b97-34bf-4546-8f35-5d378f8806d9\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") "
Jul 01 14:57:36.354386 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.354245    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-rbac-proxy-sar-config\") pod \"807a9b97-34bf-4546-8f35-5d378f8806d9\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") "
Jul 01 14:57:36.354386 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.354325    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-cabundle-cert\") pod \"807a9b97-34bf-4546-8f35-5d378f8806d9\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") "
Jul 01 14:57:36.354386 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.354354    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807a9b97-34bf-4546-8f35-5d378f8806d9-proxy-tls\") pod \"807a9b97-34bf-4546-8f35-5d378f8806d9\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") "
Jul 01 14:57:36.354515 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.354419    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/807a9b97-34bf-4546-8f35-5d378f8806d9-kserve-provision-location\") pod \"807a9b97-34bf-4546-8f35-5d378f8806d9\" (UID: \"807a9b97-34bf-4546-8f35-5d378f8806d9\") "
Jul 01 14:57:36.354808 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.354778    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807a9b97-34bf-4546-8f35-5d378f8806d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "807a9b97-34bf-4546-8f35-5d378f8806d9" (UID: "807a9b97-34bf-4546-8f35-5d378f8806d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:57:36.354808 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.354779    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "807a9b97-34bf-4546-8f35-5d378f8806d9" (UID: "807a9b97-34bf-4546-8f35-5d378f8806d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:57:36.354969 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.354805    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "807a9b97-34bf-4546-8f35-5d378f8806d9" (UID: "807a9b97-34bf-4546-8f35-5d378f8806d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:57:36.356822 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.356798    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807a9b97-34bf-4546-8f35-5d378f8806d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "807a9b97-34bf-4546-8f35-5d378f8806d9" (UID: "807a9b97-34bf-4546-8f35-5d378f8806d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:57:36.356920 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.356851    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-api-access-5kzcx" (OuterVolumeSpecName: "kube-api-access-5kzcx") pod "807a9b97-34bf-4546-8f35-5d378f8806d9" (UID: "807a9b97-34bf-4546-8f35-5d378f8806d9"). InnerVolumeSpecName "kube-api-access-5kzcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:57:36.455618 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.455517    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kzcx\" (UniqueName: \"kubernetes.io/projected/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-api-access-5kzcx\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:57:36.455825 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.455800    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:57:36.455913 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.455849    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/807a9b97-34bf-4546-8f35-5d378f8806d9-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:57:36.455913 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.455865    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807a9b97-34bf-4546-8f35-5d378f8806d9-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:57:36.455913 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:36.455886    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/807a9b97-34bf-4546-8f35-5d378f8806d9-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:57:37.138264 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.138173    2558 generic.go:356] "Generic (PLEG): container finished" podID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerID="2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337" exitCode=0
Jul 01 14:57:37.138660 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.138294    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"
Jul 01 14:57:37.138660 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.138291    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" event={"ID":"807a9b97-34bf-4546-8f35-5d378f8806d9","Type":"ContainerDied","Data":"2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337"}
Jul 01 14:57:37.138660 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.138437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m" event={"ID":"807a9b97-34bf-4546-8f35-5d378f8806d9","Type":"ContainerDied","Data":"8e8f2f8995b741fd05e9f8d2d567ac64c51f9a8808c46627b97214577dae883f"}
Jul 01 14:57:37.138660 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.138480    2558 scope.go:117] "RemoveContainer" containerID="93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20"
Jul 01 14:57:37.140577 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.140549    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" event={"ID":"c5a2cb72-1399-4e30-83aa-ccd8c293828d","Type":"ContainerStarted","Data":"e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6"}
Jul 01 14:57:37.140658 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.140617    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" event={"ID":"c5a2cb72-1399-4e30-83aa-ccd8c293828d","Type":"ContainerStarted","Data":"c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b"}
Jul 01 14:57:37.140841 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.140821    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:37.140916 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.140879    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:37.142616 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.142589    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 14:57:37.147519 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.147501    2558 scope.go:117] "RemoveContainer" containerID="2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337"
Jul 01 14:57:37.154658 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.154642    2558 scope.go:117] "RemoveContainer" containerID="2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a"
Jul 01 14:57:37.161758 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.161741    2558 scope.go:117] "RemoveContainer" containerID="93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20"
Jul 01 14:57:37.162033 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:57:37.162015    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20\": container with ID starting with 93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20 not found: ID does not exist" containerID="93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20"
Jul 01 14:57:37.162092 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.162042    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20"} err="failed to get container status \"93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20\": rpc error: code = NotFound desc = could not find container \"93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20\": container with ID starting with 93e6cbd70295a726b2c3d219efb66dbe55d30bdf14859eee05af08688725aa20 not found: ID does not exist"
Jul 01 14:57:37.162092 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.162057    2558 scope.go:117] "RemoveContainer" containerID="2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337"
Jul 01 14:57:37.162281 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:57:37.162263    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337\": container with ID starting with 2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337 not found: ID does not exist" containerID="2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337"
Jul 01 14:57:37.162326 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.162287    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337"} err="failed to get container status \"2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337\": rpc error: code = NotFound desc = could not find container \"2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337\": container with ID starting with 2aac412b93fcc11411ab7b6481fc70e5bebcb41bcc8d2e611ab4fc86c5e3b337 not found: ID does not exist"
Jul 01 14:57:37.162326 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.162303    2558 scope.go:117] "RemoveContainer" containerID="2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a"
Jul 01 14:57:37.162501 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:57:37.162486    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a\": container with ID starting with 2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a not found: ID does not exist" containerID="2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a"
Jul 01 14:57:37.162537 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.162506    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a"} err="failed to get container status \"2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a\": rpc error: code = NotFound desc = could not find container \"2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a\": container with ID starting with 2492a482e844115185affc44c00a107526c7f08bf0b32637e4aafae906f4414a not found: ID does not exist"
Jul 01 14:57:37.166731 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.166668    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podStartSLOduration=2.37934133 podStartE2EDuration="6.166655448s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.787314118s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:57:37.165322573 +0000 UTC m=+1105.225901765" watchObservedRunningTime="2026-07-01 14:57:37.166655448 +0000 UTC m=+1105.227234687"
Jul 01 14:57:37.180018 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.179985    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"]
Jul 01 14:57:37.183073 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:37.183051    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b7013-predictor-5cc5b679b8-w2g4m"]
Jul 01 14:57:38.145957 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:38.145919    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 14:57:38.353329 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:38.353295    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" path="/var/lib/kubelet/pods/807a9b97-34bf-4546-8f35-5d378f8806d9/volumes"
Jul 01 14:57:43.151025 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:43.150996    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:57:43.151529 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:43.151503    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 14:57:53.152401 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:57:53.152357    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 14:58:03.151938 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:58:03.151900    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 14:58:13.151415 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:58:13.151378    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 14:58:23.152220 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:58:23.152179    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 14:58:33.152194 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:58:33.152155    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 14:58:43.152082 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:58:43.151992    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 14:58:53.152029 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:58:53.151994    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:59:01.770363 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.770325    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"]
Jul 01 14:59:01.770769 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.770708    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kube-rbac-proxy"
Jul 01 14:59:01.770769 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.770719    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kube-rbac-proxy"
Jul 01 14:59:01.770769 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.770759    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="storage-initializer"
Jul 01 14:59:01.770769 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.770764    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="storage-initializer"
Jul 01 14:59:01.770769 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.770772    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container"
Jul 01 14:59:01.771039 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.770778    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container"
Jul 01 14:59:01.771039 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.770830    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kube-rbac-proxy"
Jul 01 14:59:01.771039 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.770838    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="807a9b97-34bf-4546-8f35-5d378f8806d9" containerName="kserve-container"
Jul 01 14:59:01.774321 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.774305    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.777049 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.777027    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-df069b\""
Jul 01 14:59:01.777171 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.777055    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-df069b-kube-rbac-proxy-sar-config\""
Jul 01 14:59:01.777171 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.777028    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-df069b-predictor-serving-cert\""
Jul 01 14:59:01.777171 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.777035    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-df069b-dockercfg-qqjwr\""
Jul 01 14:59:01.784919 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.784896    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"]
Jul 01 14:59:01.804956 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.804923    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b8d4451-d90b-4ba6-a935-916d55d1419d-proxy-tls\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.805115 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.804975    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.805115 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.805029    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d4451-d90b-4ba6-a935-916d55d1419d-kserve-provision-location\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.805115 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.805085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-cabundle-cert\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.805236 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.805128    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvlb\" (UniqueName: \"kubernetes.io/projected/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-api-access-npvlb\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.905752 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.905716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-cabundle-cert\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.905945 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.905776    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvlb\" (UniqueName: \"kubernetes.io/projected/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-api-access-npvlb\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.905945 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.905802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b8d4451-d90b-4ba6-a935-916d55d1419d-proxy-tls\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.905945 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.905836    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.905945 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.905870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d4451-d90b-4ba6-a935-916d55d1419d-kserve-provision-location\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.906332 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.906306    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d4451-d90b-4ba6-a935-916d55d1419d-kserve-provision-location\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.906465 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.906441    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-cabundle-cert\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.906515 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.906479    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.908335 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.908319    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b8d4451-d90b-4ba6-a935-916d55d1419d-proxy-tls\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:01.915716 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:01.915674    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvlb\" (UniqueName: \"kubernetes.io/projected/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-api-access-npvlb\") pod \"isvc-secondary-df069b-predictor-67966c757b-g9d6r\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") " pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:02.084222 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:02.084174    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:02.224374 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:02.224342    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"]
Jul 01 14:59:02.227494 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:59:02.227469    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8d4451_d90b_4ba6_a935_916d55d1419d.slice/crio-087fcebf027e375f70569a502c63ab7284eb903f76d16fa5f1cc3c4ee33b9bd6 WatchSource:0}: Error finding container 087fcebf027e375f70569a502c63ab7284eb903f76d16fa5f1cc3c4ee33b9bd6: Status 404 returned error can't find the container with id 087fcebf027e375f70569a502c63ab7284eb903f76d16fa5f1cc3c4ee33b9bd6
Jul 01 14:59:02.498032 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:02.497943    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r" event={"ID":"0b8d4451-d90b-4ba6-a935-916d55d1419d","Type":"ContainerStarted","Data":"c97549ce653f72b3200222b231070684637bf24bab8e089e9426c9ff4079a630"}
Jul 01 14:59:02.498032 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:02.498008    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r" event={"ID":"0b8d4451-d90b-4ba6-a935-916d55d1419d","Type":"ContainerStarted","Data":"087fcebf027e375f70569a502c63ab7284eb903f76d16fa5f1cc3c4ee33b9bd6"}
Jul 01 14:59:05.512265 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:05.512240    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-df069b-predictor-67966c757b-g9d6r_0b8d4451-d90b-4ba6-a935-916d55d1419d/storage-initializer/0.log"
Jul 01 14:59:05.512605 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:05.512278    2558 generic.go:356] "Generic (PLEG): container finished" podID="0b8d4451-d90b-4ba6-a935-916d55d1419d" containerID="c97549ce653f72b3200222b231070684637bf24bab8e089e9426c9ff4079a630" exitCode=1
Jul 01 14:59:05.512605 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:05.512305    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r" event={"ID":"0b8d4451-d90b-4ba6-a935-916d55d1419d","Type":"ContainerDied","Data":"c97549ce653f72b3200222b231070684637bf24bab8e089e9426c9ff4079a630"}
Jul 01 14:59:06.518607 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:06.518578    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-df069b-predictor-67966c757b-g9d6r_0b8d4451-d90b-4ba6-a935-916d55d1419d/storage-initializer/0.log"
Jul 01 14:59:06.518993 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:06.518713    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r" event={"ID":"0b8d4451-d90b-4ba6-a935-916d55d1419d","Type":"ContainerStarted","Data":"7bacd290a015da3e298b6178d6f59f446dd6055265a25afff2d0d5ab3c4a9f33"}
Jul 01 14:59:10.536546 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:10.536521    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-df069b-predictor-67966c757b-g9d6r_0b8d4451-d90b-4ba6-a935-916d55d1419d/storage-initializer/1.log"
Jul 01 14:59:10.537000 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:10.536980    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-df069b-predictor-67966c757b-g9d6r_0b8d4451-d90b-4ba6-a935-916d55d1419d/storage-initializer/0.log"
Jul 01 14:59:10.537072 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:10.537023    2558 generic.go:356] "Generic (PLEG): container finished" podID="0b8d4451-d90b-4ba6-a935-916d55d1419d" containerID="7bacd290a015da3e298b6178d6f59f446dd6055265a25afff2d0d5ab3c4a9f33" exitCode=1
Jul 01 14:59:10.537124 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:10.537096    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r" event={"ID":"0b8d4451-d90b-4ba6-a935-916d55d1419d","Type":"ContainerDied","Data":"7bacd290a015da3e298b6178d6f59f446dd6055265a25afff2d0d5ab3c4a9f33"}
Jul 01 14:59:10.537168 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:10.537157    2558 scope.go:117] "RemoveContainer" containerID="c97549ce653f72b3200222b231070684637bf24bab8e089e9426c9ff4079a630"
Jul 01 14:59:10.537559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:10.537543    2558 scope.go:117] "RemoveContainer" containerID="c97549ce653f72b3200222b231070684637bf24bab8e089e9426c9ff4079a630"
Jul 01 14:59:10.547275 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:59:10.547254    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-df069b-predictor-67966c757b-g9d6r_kserve-ci-e2e-test(0b8d4451-d90b-4ba6-a935-916d55d1419d)\"" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r" podUID="0b8d4451-d90b-4ba6-a935-916d55d1419d"
Jul 01 14:59:10.547406 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:59:10.547379    2558 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-df069b-predictor-67966c757b-g9d6r_kserve-ci-e2e-test_0b8d4451-d90b-4ba6-a935-916d55d1419d_0 in pod sandbox 087fcebf027e375f70569a502c63ab7284eb903f76d16fa5f1cc3c4ee33b9bd6 from index: no such id: 'c97549ce653f72b3200222b231070684637bf24bab8e089e9426c9ff4079a630'" containerID="c97549ce653f72b3200222b231070684637bf24bab8e089e9426c9ff4079a630"
Jul 01 14:59:10.547472 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:10.547414    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97549ce653f72b3200222b231070684637bf24bab8e089e9426c9ff4079a630"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-df069b-predictor-67966c757b-g9d6r_kserve-ci-e2e-test_0b8d4451-d90b-4ba6-a935-916d55d1419d_0 in pod sandbox 087fcebf027e375f70569a502c63ab7284eb903f76d16fa5f1cc3c4ee33b9bd6 from index: no such id: 'c97549ce653f72b3200222b231070684637bf24bab8e089e9426c9ff4079a630'"
Jul 01 14:59:11.542510 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:11.542480    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-df069b-predictor-67966c757b-g9d6r_0b8d4451-d90b-4ba6-a935-916d55d1419d/storage-initializer/1.log"
Jul 01 14:59:12.382071 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:12.382045    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-df069b-predictor-67966c757b-g9d6r_0b8d4451-d90b-4ba6-a935-916d55d1419d/storage-initializer/1.log"
Jul 01 14:59:12.383706 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:12.383664    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-df069b-predictor-67966c757b-g9d6r_0b8d4451-d90b-4ba6-a935-916d55d1419d/storage-initializer/1.log"
Jul 01 14:59:12.398553 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:12.398532    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:59:12.400024 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:12.400004    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 14:59:17.822493 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:17.822461    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"]
Jul 01 14:59:17.889591 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:17.889558    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"]
Jul 01 14:59:17.889848 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:17.889824    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container" containerID="cri-o://c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b" gracePeriod=30
Jul 01 14:59:17.889920 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:17.889876    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kube-rbac-proxy" containerID="cri-o://e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6" gracePeriod=30
Jul 01 14:59:18.010180 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.010159    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-df069b-predictor-67966c757b-g9d6r_0b8d4451-d90b-4ba6-a935-916d55d1419d/storage-initializer/1.log"
Jul 01 14:59:18.012296 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.012281    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:18.145426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.145348    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d4451-d90b-4ba6-a935-916d55d1419d-kserve-provision-location\") pod \"0b8d4451-d90b-4ba6-a935-916d55d1419d\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") "
Jul 01 14:59:18.145426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.145386    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b8d4451-d90b-4ba6-a935-916d55d1419d-proxy-tls\") pod \"0b8d4451-d90b-4ba6-a935-916d55d1419d\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") "
Jul 01 14:59:18.145426 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.145423    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npvlb\" (UniqueName: \"kubernetes.io/projected/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-api-access-npvlb\") pod \"0b8d4451-d90b-4ba6-a935-916d55d1419d\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") "
Jul 01 14:59:18.145721 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.145441    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-rbac-proxy-sar-config\") pod \"0b8d4451-d90b-4ba6-a935-916d55d1419d\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") "
Jul 01 14:59:18.145721 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.145464    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-cabundle-cert\") pod \"0b8d4451-d90b-4ba6-a935-916d55d1419d\" (UID: \"0b8d4451-d90b-4ba6-a935-916d55d1419d\") "
Jul 01 14:59:18.145721 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.145663    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d4451-d90b-4ba6-a935-916d55d1419d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b8d4451-d90b-4ba6-a935-916d55d1419d" (UID: "0b8d4451-d90b-4ba6-a935-916d55d1419d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:59:18.145889 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.145860    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b8d4451-d90b-4ba6-a935-916d55d1419d" (UID: "0b8d4451-d90b-4ba6-a935-916d55d1419d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:59:18.145947 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.145902    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b8d4451-d90b-4ba6-a935-916d55d1419d" (UID: "0b8d4451-d90b-4ba6-a935-916d55d1419d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:59:18.146091 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.146067    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 14:59:18.147496 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.147475    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-api-access-npvlb" (OuterVolumeSpecName: "kube-api-access-npvlb") pod "0b8d4451-d90b-4ba6-a935-916d55d1419d" (UID: "0b8d4451-d90b-4ba6-a935-916d55d1419d"). InnerVolumeSpecName "kube-api-access-npvlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:59:18.147563 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.147524    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8d4451-d90b-4ba6-a935-916d55d1419d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b8d4451-d90b-4ba6-a935-916d55d1419d" (UID: "0b8d4451-d90b-4ba6-a935-916d55d1419d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:59:18.246394 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.246366    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d4451-d90b-4ba6-a935-916d55d1419d-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:18.246394 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.246390    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b8d4451-d90b-4ba6-a935-916d55d1419d-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:18.246394 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.246400    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npvlb\" (UniqueName: \"kubernetes.io/projected/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-api-access-npvlb\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:18.246587 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.246411    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:18.246587 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.246420    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b8d4451-d90b-4ba6-a935-916d55d1419d-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:18.574311 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.574277    2558 generic.go:356] "Generic (PLEG): container finished" podID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerID="e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6" exitCode=2
Jul 01 14:59:18.574482 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.574351    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" event={"ID":"c5a2cb72-1399-4e30-83aa-ccd8c293828d","Type":"ContainerDied","Data":"e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6"}
Jul 01 14:59:18.575510 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.575494    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-df069b-predictor-67966c757b-g9d6r_0b8d4451-d90b-4ba6-a935-916d55d1419d/storage-initializer/1.log"
Jul 01 14:59:18.575641 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.575531    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r" event={"ID":"0b8d4451-d90b-4ba6-a935-916d55d1419d","Type":"ContainerDied","Data":"087fcebf027e375f70569a502c63ab7284eb903f76d16fa5f1cc3c4ee33b9bd6"}
Jul 01 14:59:18.575641 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.575575    2558 scope.go:117] "RemoveContainer" containerID="7bacd290a015da3e298b6178d6f59f446dd6055265a25afff2d0d5ab3c4a9f33"
Jul 01 14:59:18.575641 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.575583    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"
Jul 01 14:59:18.612558 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.612529    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"]
Jul 01 14:59:18.616091 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:18.616057    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-df069b-predictor-67966c757b-g9d6r"]
Jul 01 14:59:20.353674 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:20.353641    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b8d4451-d90b-4ba6-a935-916d55d1419d" path="/var/lib/kubelet/pods/0b8d4451-d90b-4ba6-a935-916d55d1419d/volumes"
Jul 01 14:59:22.554425 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.554403    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:59:22.594294 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.594256    2558 generic.go:356] "Generic (PLEG): container finished" podID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerID="c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b" exitCode=0
Jul 01 14:59:22.594435 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.594320    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"
Jul 01 14:59:22.594435 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.594340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" event={"ID":"c5a2cb72-1399-4e30-83aa-ccd8c293828d","Type":"ContainerDied","Data":"c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b"}
Jul 01 14:59:22.594435 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.594409    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh" event={"ID":"c5a2cb72-1399-4e30-83aa-ccd8c293828d","Type":"ContainerDied","Data":"9dd73586c3668982ffa8bda745d2513cc245e73058180601459cdf3bb1792882"}
Jul 01 14:59:22.594550 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.594449    2558 scope.go:117] "RemoveContainer" containerID="e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6"
Jul 01 14:59:22.605202 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.605184    2558 scope.go:117] "RemoveContainer" containerID="c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b"
Jul 01 14:59:22.612492 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.612471    2558 scope.go:117] "RemoveContainer" containerID="7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87"
Jul 01 14:59:22.619605 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.619588    2558 scope.go:117] "RemoveContainer" containerID="e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6"
Jul 01 14:59:22.619879 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:59:22.619856    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6\": container with ID starting with e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6 not found: ID does not exist" containerID="e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6"
Jul 01 14:59:22.619945 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.619896    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6"} err="failed to get container status \"e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6\": rpc error: code = NotFound desc = could not find container \"e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6\": container with ID starting with e10b2de05223ed4696381c3881498f4b6a3a1a075e6f4ba058d3f6c629c4a7d6 not found: ID does not exist"
Jul 01 14:59:22.619945 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.619916    2558 scope.go:117] "RemoveContainer" containerID="c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b"
Jul 01 14:59:22.620132 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:59:22.620117    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b\": container with ID starting with c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b not found: ID does not exist" containerID="c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b"
Jul 01 14:59:22.620176 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.620136    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b"} err="failed to get container status \"c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b\": rpc error: code = NotFound desc = could not find container \"c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b\": container with ID starting with c39c4c8cb14bcd2fa04100b596863d6749a3ceacfd61d4e137818fbb7645474b not found: ID does not exist"
Jul 01 14:59:22.620176 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.620149    2558 scope.go:117] "RemoveContainer" containerID="7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87"
Jul 01 14:59:22.620403 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:59:22.620382    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87\": container with ID starting with 7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87 not found: ID does not exist" containerID="7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87"
Jul 01 14:59:22.620450 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.620408    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87"} err="failed to get container status \"7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87\": rpc error: code = NotFound desc = could not find container \"7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87\": container with ID starting with 7e2ddc9b42943582aadf310ba82d5c2bab94111f4bfb8c7df6386d96a31b5b87 not found: ID does not exist"
Jul 01 14:59:22.682057 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.681990    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a2cb72-1399-4e30-83aa-ccd8c293828d-proxy-tls\") pod \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") "
Jul 01 14:59:22.682057 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.682033    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-rbac-proxy-sar-config\") pod \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") "
Jul 01 14:59:22.682256 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.682067    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7shdh\" (UniqueName: \"kubernetes.io/projected/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-api-access-7shdh\") pod \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") "
Jul 01 14:59:22.682256 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.682108    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-cabundle-cert\") pod \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") "
Jul 01 14:59:22.682256 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.682212    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kserve-provision-location\") pod \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\" (UID: \"c5a2cb72-1399-4e30-83aa-ccd8c293828d\") "
Jul 01 14:59:22.682559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.682529    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5a2cb72-1399-4e30-83aa-ccd8c293828d" (UID: "c5a2cb72-1399-4e30-83aa-ccd8c293828d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:59:22.682559 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.682546    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5a2cb72-1399-4e30-83aa-ccd8c293828d" (UID: "c5a2cb72-1399-4e30-83aa-ccd8c293828d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:59:22.682776 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.682553    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5a2cb72-1399-4e30-83aa-ccd8c293828d" (UID: "c5a2cb72-1399-4e30-83aa-ccd8c293828d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:59:22.684380 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.684359    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-api-access-7shdh" (OuterVolumeSpecName: "kube-api-access-7shdh") pod "c5a2cb72-1399-4e30-83aa-ccd8c293828d" (UID: "c5a2cb72-1399-4e30-83aa-ccd8c293828d"). InnerVolumeSpecName "kube-api-access-7shdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:59:22.684486 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.684410    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a2cb72-1399-4e30-83aa-ccd8c293828d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5a2cb72-1399-4e30-83aa-ccd8c293828d" (UID: "c5a2cb72-1399-4e30-83aa-ccd8c293828d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:59:22.783463 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.783436    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:22.783463 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.783461    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:22.783659 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.783472    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a2cb72-1399-4e30-83aa-ccd8c293828d-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:22.783659 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.783482    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:22.783659 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.783490    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7shdh\" (UniqueName: \"kubernetes.io/projected/c5a2cb72-1399-4e30-83aa-ccd8c293828d-kube-api-access-7shdh\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:22.919443 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.919416    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"]
Jul 01 14:59:22.926931 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:22.926905    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-df069b-predictor-8597f6d579-wrvrh"]
Jul 01 14:59:23.020842 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.020759    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"]
Jul 01 14:59:23.021165 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021148    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container"
Jul 01 14:59:23.021248 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021168    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container"
Jul 01 14:59:23.021248 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021189    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8d4451-d90b-4ba6-a935-916d55d1419d" containerName="storage-initializer"
Jul 01 14:59:23.021248 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021198    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8d4451-d90b-4ba6-a935-916d55d1419d" containerName="storage-initializer"
Jul 01 14:59:23.021248 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021210    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="storage-initializer"
Jul 01 14:59:23.021248 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021219    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="storage-initializer"
Jul 01 14:59:23.021501 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021257    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kube-rbac-proxy"
Jul 01 14:59:23.021501 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021266    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kube-rbac-proxy"
Jul 01 14:59:23.021501 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021356    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kube-rbac-proxy"
Jul 01 14:59:23.021501 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021371    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" containerName="kserve-container"
Jul 01 14:59:23.021501 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021383    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8d4451-d90b-4ba6-a935-916d55d1419d" containerName="storage-initializer"
Jul 01 14:59:23.021501 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021396    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8d4451-d90b-4ba6-a935-916d55d1419d" containerName="storage-initializer"
Jul 01 14:59:23.021501 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021472    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8d4451-d90b-4ba6-a935-916d55d1419d" containerName="storage-initializer"
Jul 01 14:59:23.021501 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.021482    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8d4451-d90b-4ba6-a935-916d55d1419d" containerName="storage-initializer"
Jul 01 14:59:23.025911 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.025891    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.029448 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.029430    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-88a6a6-predictor-serving-cert\""
Jul 01 14:59:23.029550 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.029528    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-88a6a6\""
Jul 01 14:59:23.029677 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.029658    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 14:59:23.030223 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.030210    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-88a6a6-kube-rbac-proxy-sar-config\""
Jul 01 14:59:23.030453 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.030441    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 14:59:23.030506 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.030491    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-88a6a6-dockercfg-8frg5\""
Jul 01 14:59:23.030864 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.030848    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 14:59:23.038554 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.038532    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"]
Jul 01 14:59:23.086011 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.085963    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20f84f2-25b2-47bf-a495-092acea6c5db-proxy-tls\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.086176 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.086024    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4t85\" (UniqueName: \"kubernetes.io/projected/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-api-access-r4t85\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.086176 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.086058    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.086176 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.086093    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-cabundle-cert\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.086284 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.086195    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e20f84f2-25b2-47bf-a495-092acea6c5db-kserve-provision-location\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.186912 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.186878    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e20f84f2-25b2-47bf-a495-092acea6c5db-kserve-provision-location\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.187106 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.186932    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20f84f2-25b2-47bf-a495-092acea6c5db-proxy-tls\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.187106 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.186955    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4t85\" (UniqueName: \"kubernetes.io/projected/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-api-access-r4t85\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.187106 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.186983    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.187106 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.187027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-cabundle-cert\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.187106 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:59:23.187061    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-serving-cert: secret "isvc-init-fail-88a6a6-predictor-serving-cert" not found
Jul 01 14:59:23.187379 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:59:23.187163    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e20f84f2-25b2-47bf-a495-092acea6c5db-proxy-tls podName:e20f84f2-25b2-47bf-a495-092acea6c5db nodeName:}" failed. No retries permitted until 2026-07-01 14:59:23.687142461 +0000 UTC m=+1211.747721630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e20f84f2-25b2-47bf-a495-092acea6c5db-proxy-tls") pod "isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr" (UID: "e20f84f2-25b2-47bf-a495-092acea6c5db") : secret "isvc-init-fail-88a6a6-predictor-serving-cert" not found
Jul 01 14:59:23.187379 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.187337    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e20f84f2-25b2-47bf-a495-092acea6c5db-kserve-provision-location\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.187667 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.187649    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-cabundle-cert\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.187731 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.187652    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.197438 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.197421    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4t85\" (UniqueName: \"kubernetes.io/projected/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-api-access-r4t85\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.690169 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.690133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20f84f2-25b2-47bf-a495-092acea6c5db-proxy-tls\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.692434 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.692405    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20f84f2-25b2-47bf-a495-092acea6c5db-proxy-tls\") pod \"isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") " pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:23.935750 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:23.935719    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:24.065979 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:24.065953    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"]
Jul 01 14:59:24.068844 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:59:24.068821    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20f84f2_25b2_47bf_a495_092acea6c5db.slice/crio-7e39d75aef3ca880eb3ecfe694183f02fd0834dc504dd6b650b9d0142a997bcd WatchSource:0}: Error finding container 7e39d75aef3ca880eb3ecfe694183f02fd0834dc504dd6b650b9d0142a997bcd: Status 404 returned error can't find the container with id 7e39d75aef3ca880eb3ecfe694183f02fd0834dc504dd6b650b9d0142a997bcd
Jul 01 14:59:24.354167 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:24.354138    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a2cb72-1399-4e30-83aa-ccd8c293828d" path="/var/lib/kubelet/pods/c5a2cb72-1399-4e30-83aa-ccd8c293828d/volumes"
Jul 01 14:59:24.605191 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:24.605108    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr" event={"ID":"e20f84f2-25b2-47bf-a495-092acea6c5db","Type":"ContainerStarted","Data":"961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e"}
Jul 01 14:59:24.605191 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:24.605184    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr" event={"ID":"e20f84f2-25b2-47bf-a495-092acea6c5db","Type":"ContainerStarted","Data":"7e39d75aef3ca880eb3ecfe694183f02fd0834dc504dd6b650b9d0142a997bcd"}
Jul 01 14:59:27.618975 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:27.618948    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr_e20f84f2-25b2-47bf-a495-092acea6c5db/storage-initializer/0.log"
Jul 01 14:59:27.619314 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:27.618984    2558 generic.go:356] "Generic (PLEG): container finished" podID="e20f84f2-25b2-47bf-a495-092acea6c5db" containerID="961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e" exitCode=1
Jul 01 14:59:27.619314 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:27.619026    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr" event={"ID":"e20f84f2-25b2-47bf-a495-092acea6c5db","Type":"ContainerDied","Data":"961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e"}
Jul 01 14:59:28.006416 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:28.006340    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"]
Jul 01 14:59:28.624371 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:28.624342    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr_e20f84f2-25b2-47bf-a495-092acea6c5db/storage-initializer/0.log"
Jul 01 14:59:28.624836 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:28.624403    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr" event={"ID":"e20f84f2-25b2-47bf-a495-092acea6c5db","Type":"ContainerStarted","Data":"b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181"}
Jul 01 14:59:28.624836 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:28.624510    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr" podUID="e20f84f2-25b2-47bf-a495-092acea6c5db" containerName="storage-initializer" containerID="cri-o://b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181" gracePeriod=30
Jul 01 14:59:32.480308 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.480281    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr_e20f84f2-25b2-47bf-a495-092acea6c5db/storage-initializer/1.log"
Jul 01 14:59:32.480666 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.480651    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr_e20f84f2-25b2-47bf-a495-092acea6c5db/storage-initializer/0.log"
Jul 01 14:59:32.482941 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.482926    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:32.563796 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.563678    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4t85\" (UniqueName: \"kubernetes.io/projected/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-api-access-r4t85\") pod \"e20f84f2-25b2-47bf-a495-092acea6c5db\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") "
Jul 01 14:59:32.563796 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.563778    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-cabundle-cert\") pod \"e20f84f2-25b2-47bf-a495-092acea6c5db\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") "
Jul 01 14:59:32.563796 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.563795    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e20f84f2-25b2-47bf-a495-092acea6c5db-kserve-provision-location\") pod \"e20f84f2-25b2-47bf-a495-092acea6c5db\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") "
Jul 01 14:59:32.564076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.563825    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-rbac-proxy-sar-config\") pod \"e20f84f2-25b2-47bf-a495-092acea6c5db\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") "
Jul 01 14:59:32.564076 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.563867    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20f84f2-25b2-47bf-a495-092acea6c5db-proxy-tls\") pod \"e20f84f2-25b2-47bf-a495-092acea6c5db\" (UID: \"e20f84f2-25b2-47bf-a495-092acea6c5db\") "
Jul 01 14:59:32.564188 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.564112    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20f84f2-25b2-47bf-a495-092acea6c5db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e20f84f2-25b2-47bf-a495-092acea6c5db" (UID: "e20f84f2-25b2-47bf-a495-092acea6c5db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:59:32.564188 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.564181    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e20f84f2-25b2-47bf-a495-092acea6c5db" (UID: "e20f84f2-25b2-47bf-a495-092acea6c5db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:59:32.564276 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.564211    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e20f84f2-25b2-47bf-a495-092acea6c5db" (UID: "e20f84f2-25b2-47bf-a495-092acea6c5db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:59:32.565878 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.565853    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-api-access-r4t85" (OuterVolumeSpecName: "kube-api-access-r4t85") pod "e20f84f2-25b2-47bf-a495-092acea6c5db" (UID: "e20f84f2-25b2-47bf-a495-092acea6c5db"). InnerVolumeSpecName "kube-api-access-r4t85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:59:32.565959 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.565875    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e20f84f2-25b2-47bf-a495-092acea6c5db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e20f84f2-25b2-47bf-a495-092acea6c5db" (UID: "e20f84f2-25b2-47bf-a495-092acea6c5db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:59:32.643589 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.643566    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr_e20f84f2-25b2-47bf-a495-092acea6c5db/storage-initializer/1.log"
Jul 01 14:59:32.643971 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.643955    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr_e20f84f2-25b2-47bf-a495-092acea6c5db/storage-initializer/0.log"
Jul 01 14:59:32.644029 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.643991    2558 generic.go:356] "Generic (PLEG): container finished" podID="e20f84f2-25b2-47bf-a495-092acea6c5db" containerID="b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181" exitCode=1
Jul 01 14:59:32.644067 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.644050    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"
Jul 01 14:59:32.644110 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.644089    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr" event={"ID":"e20f84f2-25b2-47bf-a495-092acea6c5db","Type":"ContainerDied","Data":"b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181"}
Jul 01 14:59:32.644169 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.644154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr" event={"ID":"e20f84f2-25b2-47bf-a495-092acea6c5db","Type":"ContainerDied","Data":"7e39d75aef3ca880eb3ecfe694183f02fd0834dc504dd6b650b9d0142a997bcd"}
Jul 01 14:59:32.644202 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.644189    2558 scope.go:117] "RemoveContainer" containerID="b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181"
Jul 01 14:59:32.652876 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.652860    2558 scope.go:117] "RemoveContainer" containerID="961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e"
Jul 01 14:59:32.659998 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.659983    2558 scope.go:117] "RemoveContainer" containerID="b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181"
Jul 01 14:59:32.660247 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:59:32.660230    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181\": container with ID starting with b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181 not found: ID does not exist" containerID="b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181"
Jul 01 14:59:32.660293 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.660255    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181"} err="failed to get container status \"b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181\": rpc error: code = NotFound desc = could not find container \"b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181\": container with ID starting with b7274890bcaa59bb892df22ce785f4cd0eae7c31e94db00fe38239b8ca467181 not found: ID does not exist"
Jul 01 14:59:32.660293 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.660271    2558 scope.go:117] "RemoveContainer" containerID="961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e"
Jul 01 14:59:32.660504 ip-10-0-134-20 kubenswrapper[2558]: E0701 14:59:32.660487    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e\": container with ID starting with 961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e not found: ID does not exist" containerID="961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e"
Jul 01 14:59:32.660550 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.660509    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e"} err="failed to get container status \"961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e\": rpc error: code = NotFound desc = could not find container \"961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e\": container with ID starting with 961376e7e83154833d42084248777c177de86b6b317d24e3c2bd545a7738912e not found: ID does not exist"
Jul 01 14:59:32.664967 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.664937    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4t85\" (UniqueName: \"kubernetes.io/projected/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-api-access-r4t85\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:32.664967 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.664970    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:32.665079 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.664980    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e20f84f2-25b2-47bf-a495-092acea6c5db-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:32.665079 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.664988    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e20f84f2-25b2-47bf-a495-092acea6c5db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:32.665079 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.664997    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20f84f2-25b2-47bf-a495-092acea6c5db-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 14:59:32.684278 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.684253    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"]
Jul 01 14:59:32.689471 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:32.689204    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-88a6a6-predictor-67f959bc6b-4g4hr"]
Jul 01 14:59:33.190576 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.190538    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"]
Jul 01 14:59:33.190906 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.190893    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20f84f2-25b2-47bf-a495-092acea6c5db" containerName="storage-initializer"
Jul 01 14:59:33.190962 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.190907    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20f84f2-25b2-47bf-a495-092acea6c5db" containerName="storage-initializer"
Jul 01 14:59:33.190962 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.190917    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20f84f2-25b2-47bf-a495-092acea6c5db" containerName="storage-initializer"
Jul 01 14:59:33.190962 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.190923    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20f84f2-25b2-47bf-a495-092acea6c5db" containerName="storage-initializer"
Jul 01 14:59:33.191052 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.190971    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20f84f2-25b2-47bf-a495-092acea6c5db" containerName="storage-initializer"
Jul 01 14:59:33.191086 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.191071    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20f84f2-25b2-47bf-a495-092acea6c5db" containerName="storage-initializer"
Jul 01 14:59:33.195348 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.195329    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.197903 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.197876    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fd58b-predictor-serving-cert\""
Jul 01 14:59:33.198057 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.197955    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 14:59:33.198125 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.198110    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 14:59:33.198942 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.198925    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 14:59:33.199059 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.198943    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-n79zh\""
Jul 01 14:59:33.199059 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.198957    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fd58b-kube-rbac-proxy-sar-config\""
Jul 01 14:59:33.199059 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.198980    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 14:59:33.205337 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.205316    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"]
Jul 01 14:59:33.270760 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.270730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhhn2\" (UniqueName: \"kubernetes.io/projected/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-api-access-nhhn2\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.270892 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.270781    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kserve-provision-location\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.270892 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.270843    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.270892 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.270885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-cabundle-cert\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.270992 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.270913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e83134e2-d1de-4a5a-95ca-f1441788fc7f-proxy-tls\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.371561 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.371522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhhn2\" (UniqueName: \"kubernetes.io/projected/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-api-access-nhhn2\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.371729 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.371606    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kserve-provision-location\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.371729 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.371643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.371729 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.371708    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-cabundle-cert\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.371920 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.371745    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e83134e2-d1de-4a5a-95ca-f1441788fc7f-proxy-tls\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.372347 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.372320    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kserve-provision-location\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.372547 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.372528    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.372677 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.372659    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-cabundle-cert\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.374263 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.374239    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e83134e2-d1de-4a5a-95ca-f1441788fc7f-proxy-tls\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.381147 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.381126    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhhn2\" (UniqueName: \"kubernetes.io/projected/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-api-access-nhhn2\") pod \"raw-sklearn-fd58b-predictor-55b484874b-pdv4j\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") " pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.507374 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.507299    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:33.644381 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:33.644350    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"]
Jul 01 14:59:33.645768 ip-10-0-134-20 kubenswrapper[2558]: W0701 14:59:33.645738    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83134e2_d1de_4a5a_95ca_f1441788fc7f.slice/crio-d83c2083bf946211c4c345673b5e4e2471efe7e29ca4e110b35813598b4413c3 WatchSource:0}: Error finding container d83c2083bf946211c4c345673b5e4e2471efe7e29ca4e110b35813598b4413c3: Status 404 returned error can't find the container with id d83c2083bf946211c4c345673b5e4e2471efe7e29ca4e110b35813598b4413c3
Jul 01 14:59:34.353250 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:34.353217    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e20f84f2-25b2-47bf-a495-092acea6c5db" path="/var/lib/kubelet/pods/e20f84f2-25b2-47bf-a495-092acea6c5db/volumes"
Jul 01 14:59:34.655423 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:34.655331    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" event={"ID":"e83134e2-d1de-4a5a-95ca-f1441788fc7f","Type":"ContainerStarted","Data":"826677443cb671ba27e979be01c5dea969ddd93e181134870758d79c86b56f71"}
Jul 01 14:59:34.655423 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:34.655404    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" event={"ID":"e83134e2-d1de-4a5a-95ca-f1441788fc7f","Type":"ContainerStarted","Data":"d83c2083bf946211c4c345673b5e4e2471efe7e29ca4e110b35813598b4413c3"}
Jul 01 14:59:37.668986 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:37.668950    2558 generic.go:356] "Generic (PLEG): container finished" podID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerID="826677443cb671ba27e979be01c5dea969ddd93e181134870758d79c86b56f71" exitCode=0
Jul 01 14:59:37.669355 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:37.668997    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" event={"ID":"e83134e2-d1de-4a5a-95ca-f1441788fc7f","Type":"ContainerDied","Data":"826677443cb671ba27e979be01c5dea969ddd93e181134870758d79c86b56f71"}
Jul 01 14:59:38.675329 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:38.675296    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" event={"ID":"e83134e2-d1de-4a5a-95ca-f1441788fc7f","Type":"ContainerStarted","Data":"50fabe2f6bef1e388a102f61b24385aa50082c8f45cbdf1bb90fe82c60bb1b06"}
Jul 01 14:59:38.675752 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:38.675355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" event={"ID":"e83134e2-d1de-4a5a-95ca-f1441788fc7f","Type":"ContainerStarted","Data":"bbd2f0019a230c3c73b61784b3cf0d15a1954557e69a2a9f4ca8f61fab1291cb"}
Jul 01 14:59:38.675752 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:38.675539    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:38.700101 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:38.700058    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podStartSLOduration=1.9404102920000001 podStartE2EDuration="5.700043737s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.759633445s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:59:38.698402676 +0000 UTC m=+1226.758981880" watchObservedRunningTime="2026-07-01 14:59:38.700043737 +0000 UTC m=+1226.760622927"
Jul 01 14:59:39.680329 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:39.680292    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:39.681552 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:39.681526    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 14:59:40.684452 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:40.684408    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 14:59:45.690568 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:45.690531    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 14:59:45.691202 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:45.691177    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 14:59:55.691312 ip-10-0-134-20 kubenswrapper[2558]: I0701 14:59:55.691276    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 15:00:05.691468 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:00:05.691388    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 15:00:15.691538 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:00:15.691497    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 15:00:25.691173 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:00:25.691132    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 15:00:35.692100 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:00:35.692058    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 15:00:45.691814 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:00:45.691777    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 15:00:55.692704 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:00:55.692656    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 15:01:03.387016 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:03.386981    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"]
Jul 01 15:01:03.387421 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:03.387246    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" containerID="cri-o://bbd2f0019a230c3c73b61784b3cf0d15a1954557e69a2a9f4ca8f61fab1291cb" gracePeriod=30
Jul 01 15:01:03.387421 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:03.387269    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kube-rbac-proxy" containerID="cri-o://50fabe2f6bef1e388a102f61b24385aa50082c8f45cbdf1bb90fe82c60bb1b06" gracePeriod=30
Jul 01 15:01:04.027364 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:04.027331    2558 generic.go:356] "Generic (PLEG): container finished" podID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerID="50fabe2f6bef1e388a102f61b24385aa50082c8f45cbdf1bb90fe82c60bb1b06" exitCode=2
Jul 01 15:01:04.027364 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:04.027368    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" event={"ID":"e83134e2-d1de-4a5a-95ca-f1441788fc7f","Type":"ContainerDied","Data":"50fabe2f6bef1e388a102f61b24385aa50082c8f45cbdf1bb90fe82c60bb1b06"}
Jul 01 15:01:05.685228 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:05.685171    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 01 15:01:05.691675 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:05.691630    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 15:01:08.045489 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.045461    2558 generic.go:356] "Generic (PLEG): container finished" podID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerID="bbd2f0019a230c3c73b61784b3cf0d15a1954557e69a2a9f4ca8f61fab1291cb" exitCode=0
Jul 01 15:01:08.045796 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.045524    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" event={"ID":"e83134e2-d1de-4a5a-95ca-f1441788fc7f","Type":"ContainerDied","Data":"bbd2f0019a230c3c73b61784b3cf0d15a1954557e69a2a9f4ca8f61fab1291cb"}
Jul 01 15:01:08.045796 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.045576    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j" event={"ID":"e83134e2-d1de-4a5a-95ca-f1441788fc7f","Type":"ContainerDied","Data":"d83c2083bf946211c4c345673b5e4e2471efe7e29ca4e110b35813598b4413c3"}
Jul 01 15:01:08.045796 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.045597    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d83c2083bf946211c4c345673b5e4e2471efe7e29ca4e110b35813598b4413c3"
Jul 01 15:01:08.045903 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.045809    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 15:01:08.162928 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.162846    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-cabundle-cert\") pod \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") "
Jul 01 15:01:08.162928 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.162904    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e83134e2-d1de-4a5a-95ca-f1441788fc7f-proxy-tls\") pod \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") "
Jul 01 15:01:08.163158 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.162941    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kserve-provision-location\") pod \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") "
Jul 01 15:01:08.163158 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.162972    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-rbac-proxy-sar-config\") pod \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") "
Jul 01 15:01:08.163158 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.163015    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhhn2\" (UniqueName: \"kubernetes.io/projected/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-api-access-nhhn2\") pod \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\" (UID: \"e83134e2-d1de-4a5a-95ca-f1441788fc7f\") "
Jul 01 15:01:08.163324 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.163279    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e83134e2-d1de-4a5a-95ca-f1441788fc7f" (UID: "e83134e2-d1de-4a5a-95ca-f1441788fc7f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 15:01:08.163324 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.163304    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e83134e2-d1de-4a5a-95ca-f1441788fc7f" (UID: "e83134e2-d1de-4a5a-95ca-f1441788fc7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 15:01:08.163418 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.163344    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e83134e2-d1de-4a5a-95ca-f1441788fc7f" (UID: "e83134e2-d1de-4a5a-95ca-f1441788fc7f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 15:01:08.164985 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.164962    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-api-access-nhhn2" (OuterVolumeSpecName: "kube-api-access-nhhn2") pod "e83134e2-d1de-4a5a-95ca-f1441788fc7f" (UID: "e83134e2-d1de-4a5a-95ca-f1441788fc7f"). InnerVolumeSpecName "kube-api-access-nhhn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 15:01:08.165092 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.165074    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e83134e2-d1de-4a5a-95ca-f1441788fc7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e83134e2-d1de-4a5a-95ca-f1441788fc7f" (UID: "e83134e2-d1de-4a5a-95ca-f1441788fc7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 15:01:08.264392 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.264359    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:01:08.264392 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.264385    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e83134e2-d1de-4a5a-95ca-f1441788fc7f-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:01:08.264392 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.264395    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:01:08.264618 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.264405    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:01:08.264618 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.264414    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhhn2\" (UniqueName: \"kubernetes.io/projected/e83134e2-d1de-4a5a-95ca-f1441788fc7f-kube-api-access-nhhn2\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:01:08.496232 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.496153    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"]
Jul 01 15:01:08.496486 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.496473    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="storage-initializer"
Jul 01 15:01:08.496532 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.496487    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="storage-initializer"
Jul 01 15:01:08.496532 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.496511    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kube-rbac-proxy"
Jul 01 15:01:08.496532 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.496516    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kube-rbac-proxy"
Jul 01 15:01:08.496532 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.496524    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container"
Jul 01 15:01:08.496532 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.496531    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container"
Jul 01 15:01:08.496707 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.496579    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kserve-container"
Jul 01 15:01:08.496707 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.496588    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" containerName="kube-rbac-proxy"
Jul 01 15:01:08.499743 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.499726    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.502126 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.502104    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-59568-kube-rbac-proxy-sar-config\""
Jul 01 15:01:08.502245 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.502207    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-59568-predictor-serving-cert\""
Jul 01 15:01:08.510432 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.510407    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"]
Jul 01 15:01:08.567810 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.567772    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-cabundle-cert\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.567988 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.567816    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctv7t\" (UniqueName: \"kubernetes.io/projected/86300943-7406-4dd0-b543-f96086127c11-kube-api-access-ctv7t\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.567988 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.567859    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86300943-7406-4dd0-b543-f96086127c11-kserve-provision-location\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.567988 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.567893    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86300943-7406-4dd0-b543-f96086127c11-proxy-tls\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.568104 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.568026    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.669044 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.669004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.669217 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.669054    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-cabundle-cert\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.669217 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.669168    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctv7t\" (UniqueName: \"kubernetes.io/projected/86300943-7406-4dd0-b543-f96086127c11-kube-api-access-ctv7t\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.669217 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.669209    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86300943-7406-4dd0-b543-f96086127c11-kserve-provision-location\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.669395 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.669234    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86300943-7406-4dd0-b543-f96086127c11-proxy-tls\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.669456 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:01:08.669441    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-serving-cert: secret "raw-sklearn-runtime-59568-predictor-serving-cert" not found
Jul 01 15:01:08.669512 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:01:08.669498    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86300943-7406-4dd0-b543-f96086127c11-proxy-tls podName:86300943-7406-4dd0-b543-f96086127c11 nodeName:}" failed. No retries permitted until 2026-07-01 15:01:09.169484324 +0000 UTC m=+1317.230063497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/86300943-7406-4dd0-b543-f96086127c11-proxy-tls") pod "raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" (UID: "86300943-7406-4dd0-b543-f96086127c11") : secret "raw-sklearn-runtime-59568-predictor-serving-cert" not found
Jul 01 15:01:08.669610 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.669589    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86300943-7406-4dd0-b543-f96086127c11-kserve-provision-location\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.669759 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.669742    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.669805 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.669779    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-cabundle-cert\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:08.678340 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:08.678317    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctv7t\" (UniqueName: \"kubernetes.io/projected/86300943-7406-4dd0-b543-f96086127c11-kube-api-access-ctv7t\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:09.049134 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:09.049096    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"
Jul 01 15:01:09.072555 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:09.072518    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"]
Jul 01 15:01:09.077324 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:09.077302    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fd58b-predictor-55b484874b-pdv4j"]
Jul 01 15:01:09.173549 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:09.173518    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86300943-7406-4dd0-b543-f96086127c11-proxy-tls\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:09.175869 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:09.175847    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86300943-7406-4dd0-b543-f96086127c11-proxy-tls\") pod \"raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:09.409728 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:09.409680    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:09.540136 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:09.540110    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"]
Jul 01 15:01:09.542208 ip-10-0-134-20 kubenswrapper[2558]: W0701 15:01:09.542183    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86300943_7406_4dd0_b543_f96086127c11.slice/crio-2285814619ce974861a7451deeb0af3342e316986d562dbe80bb2d67aabe363c WatchSource:0}: Error finding container 2285814619ce974861a7451deeb0af3342e316986d562dbe80bb2d67aabe363c: Status 404 returned error can't find the container with id 2285814619ce974861a7451deeb0af3342e316986d562dbe80bb2d67aabe363c
Jul 01 15:01:10.054276 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:10.054240    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" event={"ID":"86300943-7406-4dd0-b543-f96086127c11","Type":"ContainerStarted","Data":"4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439"}
Jul 01 15:01:10.054633 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:10.054313    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" event={"ID":"86300943-7406-4dd0-b543-f96086127c11","Type":"ContainerStarted","Data":"2285814619ce974861a7451deeb0af3342e316986d562dbe80bb2d67aabe363c"}
Jul 01 15:01:10.353508 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:10.353481    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e83134e2-d1de-4a5a-95ca-f1441788fc7f" path="/var/lib/kubelet/pods/e83134e2-d1de-4a5a-95ca-f1441788fc7f/volumes"
Jul 01 15:01:14.072224 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:14.072192    2558 generic.go:356] "Generic (PLEG): container finished" podID="86300943-7406-4dd0-b543-f96086127c11" containerID="4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439" exitCode=0
Jul 01 15:01:14.072632 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:14.072247    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" event={"ID":"86300943-7406-4dd0-b543-f96086127c11","Type":"ContainerDied","Data":"4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439"}
Jul 01 15:01:15.078789 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:15.078760    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" event={"ID":"86300943-7406-4dd0-b543-f96086127c11","Type":"ContainerStarted","Data":"7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984"}
Jul 01 15:01:15.079158 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:15.078816    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" event={"ID":"86300943-7406-4dd0-b543-f96086127c11","Type":"ContainerStarted","Data":"3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca"}
Jul 01 15:01:15.079158 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:15.078993    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:15.108471 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:15.108396    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podStartSLOduration=3.475544245 podStartE2EDuration="7.108380578s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.632836333s" isStatefulPod=false podCreationTimestamp="2026-07-01 15:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 15:01:15.107989555 +0000 UTC m=+1323.168568735" watchObservedRunningTime="2026-07-01 15:01:15.108380578 +0000 UTC m=+1323.168959768"
Jul 01 15:01:16.083273 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:16.083242    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:16.084510 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:16.084481    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 15:01:17.088030 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:17.087977    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 15:01:22.092359 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:22.092328    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:01:22.092893 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:22.092861    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 15:01:32.092860 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:32.092819    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 15:01:42.093746 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:42.093655    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 15:01:52.092944 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:01:52.092906    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 15:02:02.093120 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:02.093076    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 15:02:12.093301 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:12.093256    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 15:02:22.093839 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:22.093799    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 15:02:32.093543 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:32.093515    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:02:38.630133 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:38.630095    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"]
Jul 01 15:02:38.630625 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:38.630406    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" containerID="cri-o://3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca" gracePeriod=30
Jul 01 15:02:38.630625 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:38.630471    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kube-rbac-proxy" containerID="cri-o://7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984" gracePeriod=30
Jul 01 15:02:39.441824 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:39.441788    2558 generic.go:356] "Generic (PLEG): container finished" podID="86300943-7406-4dd0-b543-f96086127c11" containerID="7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984" exitCode=2
Jul 01 15:02:39.441995 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:39.441861    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" event={"ID":"86300943-7406-4dd0-b543-f96086127c11","Type":"ContainerDied","Data":"7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984"}
Jul 01 15:02:42.088794 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:42.088752    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 01 15:02:42.093031 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:42.093002    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 15:02:43.273852 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.273827    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:02:43.395292 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.395216    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86300943-7406-4dd0-b543-f96086127c11-kserve-provision-location\") pod \"86300943-7406-4dd0-b543-f96086127c11\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") "
Jul 01 15:02:43.395292 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.395252    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86300943-7406-4dd0-b543-f96086127c11-proxy-tls\") pod \"86300943-7406-4dd0-b543-f96086127c11\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") "
Jul 01 15:02:43.395485 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.395297    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-kube-rbac-proxy-sar-config\") pod \"86300943-7406-4dd0-b543-f96086127c11\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") "
Jul 01 15:02:43.395485 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.395333    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-cabundle-cert\") pod \"86300943-7406-4dd0-b543-f96086127c11\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") "
Jul 01 15:02:43.395485 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.395351    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctv7t\" (UniqueName: \"kubernetes.io/projected/86300943-7406-4dd0-b543-f96086127c11-kube-api-access-ctv7t\") pod \"86300943-7406-4dd0-b543-f96086127c11\" (UID: \"86300943-7406-4dd0-b543-f96086127c11\") "
Jul 01 15:02:43.395655 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.395625    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86300943-7406-4dd0-b543-f96086127c11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86300943-7406-4dd0-b543-f96086127c11" (UID: "86300943-7406-4dd0-b543-f96086127c11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 15:02:43.395747 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.395639    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86300943-7406-4dd0-b543-f96086127c11" (UID: "86300943-7406-4dd0-b543-f96086127c11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 15:02:43.395747 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.395679    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86300943-7406-4dd0-b543-f96086127c11" (UID: "86300943-7406-4dd0-b543-f96086127c11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 15:02:43.397381 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.397354    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86300943-7406-4dd0-b543-f96086127c11-kube-api-access-ctv7t" (OuterVolumeSpecName: "kube-api-access-ctv7t") pod "86300943-7406-4dd0-b543-f96086127c11" (UID: "86300943-7406-4dd0-b543-f96086127c11"). InnerVolumeSpecName "kube-api-access-ctv7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 15:02:43.397486 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.397427    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86300943-7406-4dd0-b543-f96086127c11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86300943-7406-4dd0-b543-f96086127c11" (UID: "86300943-7406-4dd0-b543-f96086127c11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 15:02:43.459708 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.459658    2558 generic.go:356] "Generic (PLEG): container finished" podID="86300943-7406-4dd0-b543-f96086127c11" containerID="3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca" exitCode=0
Jul 01 15:02:43.459868 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.459731    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" event={"ID":"86300943-7406-4dd0-b543-f96086127c11","Type":"ContainerDied","Data":"3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca"}
Jul 01 15:02:43.459868 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.459746    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"
Jul 01 15:02:43.459868 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.459791    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9" event={"ID":"86300943-7406-4dd0-b543-f96086127c11","Type":"ContainerDied","Data":"2285814619ce974861a7451deeb0af3342e316986d562dbe80bb2d67aabe363c"}
Jul 01 15:02:43.459868 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.459830    2558 scope.go:117] "RemoveContainer" containerID="7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984"
Jul 01 15:02:43.468654 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.468637    2558 scope.go:117] "RemoveContainer" containerID="3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca"
Jul 01 15:02:43.475721 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.475701    2558 scope.go:117] "RemoveContainer" containerID="4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439"
Jul 01 15:02:43.482477 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.482455    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"]
Jul 01 15:02:43.483290 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.483275    2558 scope.go:117] "RemoveContainer" containerID="7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984"
Jul 01 15:02:43.483544 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:02:43.483525    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984\": container with ID starting with 7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984 not found: ID does not exist" containerID="7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984"
Jul 01 15:02:43.483598 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.483554    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984"} err="failed to get container status \"7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984\": rpc error: code = NotFound desc = could not find container \"7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984\": container with ID starting with 7aeff5d58289e2180ecaabfe86233750eb727e1de9b07360d7426b4ada5f5984 not found: ID does not exist"
Jul 01 15:02:43.483598 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.483571    2558 scope.go:117] "RemoveContainer" containerID="3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca"
Jul 01 15:02:43.483830 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:02:43.483806    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca\": container with ID starting with 3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca not found: ID does not exist" containerID="3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca"
Jul 01 15:02:43.483830 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.483826    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca"} err="failed to get container status \"3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca\": rpc error: code = NotFound desc = could not find container \"3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca\": container with ID starting with 3fcfa098d83116ae6a5a40254218d11836d3cd9e0892550e223a11498ac0ebca not found: ID does not exist"
Jul 01 15:02:43.483934 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.483838    2558 scope.go:117] "RemoveContainer" containerID="4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439"
Jul 01 15:02:43.484069 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:02:43.484050    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439\": container with ID starting with 4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439 not found: ID does not exist" containerID="4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439"
Jul 01 15:02:43.484106 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.484076    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439"} err="failed to get container status \"4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439\": rpc error: code = NotFound desc = could not find container \"4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439\": container with ID starting with 4d44b1f5f36a58d34510c2da6d9750cf2c5a8937d38f5eddafb4173c9e6d2439 not found: ID does not exist"
Jul 01 15:02:43.486827 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.486808    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-59568-predictor-6849b9cc8b-7crw9"]
Jul 01 15:02:43.495964 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.495946    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-cabundle-cert\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:02:43.496049 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.495967    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctv7t\" (UniqueName: \"kubernetes.io/projected/86300943-7406-4dd0-b543-f96086127c11-kube-api-access-ctv7t\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:02:43.496049 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.495975    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86300943-7406-4dd0-b543-f96086127c11-kserve-provision-location\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:02:43.496049 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.495984    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86300943-7406-4dd0-b543-f96086127c11-proxy-tls\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:02:43.496049 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:43.495996    2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86300943-7406-4dd0-b543-f96086127c11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:02:44.353749 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.353715    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86300943-7406-4dd0-b543-f96086127c11" path="/var/lib/kubelet/pods/86300943-7406-4dd0-b543-f96086127c11/volumes"
Jul 01 15:02:44.635592 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.635514    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pp527/must-gather-hvfxm"]
Jul 01 15:02:44.635848 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.635836    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kube-rbac-proxy"
Jul 01 15:02:44.635918 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.635849    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kube-rbac-proxy"
Jul 01 15:02:44.635918 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.635868    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="storage-initializer"
Jul 01 15:02:44.635918 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.635873    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="storage-initializer"
Jul 01 15:02:44.635918 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.635890    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container"
Jul 01 15:02:44.635918 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.635897    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container"
Jul 01 15:02:44.636090 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.635943    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kube-rbac-proxy"
Jul 01 15:02:44.636090 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.635955    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="86300943-7406-4dd0-b543-f96086127c11" containerName="kserve-container"
Jul 01 15:02:44.639914 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.639893    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:02:44.642669 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.642640    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pp527\"/\"openshift-service-ca.crt\""
Jul 01 15:02:44.643642 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.643627    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pp527\"/\"default-dockercfg-w5w7q\""
Jul 01 15:02:44.643760 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.643656    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pp527\"/\"kube-root-ca.crt\""
Jul 01 15:02:44.646833 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.646812    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pp527/must-gather-hvfxm"]
Jul 01 15:02:44.706236 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.706206    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8f8a310-4471-44da-be48-de71ce7a5798-must-gather-output\") pod \"must-gather-hvfxm\" (UID: \"e8f8a310-4471-44da-be48-de71ce7a5798\") " pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:02:44.706350 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.706261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbbc\" (UniqueName: \"kubernetes.io/projected/e8f8a310-4471-44da-be48-de71ce7a5798-kube-api-access-rxbbc\") pod \"must-gather-hvfxm\" (UID: \"e8f8a310-4471-44da-be48-de71ce7a5798\") " pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:02:44.807393 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.807366    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8f8a310-4471-44da-be48-de71ce7a5798-must-gather-output\") pod \"must-gather-hvfxm\" (UID: \"e8f8a310-4471-44da-be48-de71ce7a5798\") " pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:02:44.807525 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.807422    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbbc\" (UniqueName: \"kubernetes.io/projected/e8f8a310-4471-44da-be48-de71ce7a5798-kube-api-access-rxbbc\") pod \"must-gather-hvfxm\" (UID: \"e8f8a310-4471-44da-be48-de71ce7a5798\") " pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:02:44.807676 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.807659    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8f8a310-4471-44da-be48-de71ce7a5798-must-gather-output\") pod \"must-gather-hvfxm\" (UID: \"e8f8a310-4471-44da-be48-de71ce7a5798\") " pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:02:44.816389 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.816369    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbbc\" (UniqueName: \"kubernetes.io/projected/e8f8a310-4471-44da-be48-de71ce7a5798-kube-api-access-rxbbc\") pod \"must-gather-hvfxm\" (UID: \"e8f8a310-4471-44da-be48-de71ce7a5798\") " pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:02:44.958408 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:44.958319    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:02:45.088622 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:45.088598    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pp527/must-gather-hvfxm"]
Jul 01 15:02:45.094122 ip-10-0-134-20 kubenswrapper[2558]: W0701 15:02:45.093968    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8f8a310_4471_44da_be48_de71ce7a5798.slice/crio-53e9b6a6fa19bca2825bc1cc04a72cccfed0f74c531d448901fd9f2043b48f08 WatchSource:0}: Error finding container 53e9b6a6fa19bca2825bc1cc04a72cccfed0f74c531d448901fd9f2043b48f08: Status 404 returned error can't find the container with id 53e9b6a6fa19bca2825bc1cc04a72cccfed0f74c531d448901fd9f2043b48f08
Jul 01 15:02:45.095460 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:45.095445    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 15:02:45.475937 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:45.475903    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp527/must-gather-hvfxm" event={"ID":"e8f8a310-4471-44da-be48-de71ce7a5798","Type":"ContainerStarted","Data":"53e9b6a6fa19bca2825bc1cc04a72cccfed0f74c531d448901fd9f2043b48f08"}
Jul 01 15:02:51.508845 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:51.508753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp527/must-gather-hvfxm" event={"ID":"e8f8a310-4471-44da-be48-de71ce7a5798","Type":"ContainerStarted","Data":"cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214"}
Jul 01 15:02:51.508845 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:51.508796    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp527/must-gather-hvfxm" event={"ID":"e8f8a310-4471-44da-be48-de71ce7a5798","Type":"ContainerStarted","Data":"a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4"}
Jul 01 15:02:51.526046 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:02:51.525992    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pp527/must-gather-hvfxm" podStartSLOduration=1.5268868850000001 podStartE2EDuration="7.525975078s" totalImagesPullingTime="5.999088193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 15:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 15:02:51.524441414 +0000 UTC m=+1419.585020605" watchObservedRunningTime="2026-07-01 15:02:51.525975078 +0000 UTC m=+1419.586554269"
Jul 01 15:03:12.604308 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:12.604277    2558 generic.go:356] "Generic (PLEG): container finished" podID="e8f8a310-4471-44da-be48-de71ce7a5798" containerID="a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4" exitCode=0
Jul 01 15:03:12.604823 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:12.604332    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp527/must-gather-hvfxm" event={"ID":"e8f8a310-4471-44da-be48-de71ce7a5798","Type":"ContainerDied","Data":"a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4"}
Jul 01 15:03:12.604823 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:12.604747    2558 scope.go:117] "RemoveContainer" containerID="a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4"
Jul 01 15:03:13.541871 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:13.541841    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-pp527_must-gather-hvfxm_e8f8a310-4471-44da-be48-de71ce7a5798/gather/0.log"
Jul 01 15:03:14.090277 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.090244    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k9tvz/must-gather-l9j2j"]
Jul 01 15:03:14.092777 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.092753    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9tvz/must-gather-l9j2j"
Jul 01 15:03:14.095355 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.095335    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k9tvz\"/\"default-dockercfg-gj2kd\""
Jul 01 15:03:14.095458 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.095375    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k9tvz\"/\"openshift-service-ca.crt\""
Jul 01 15:03:14.095458 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.095378    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k9tvz\"/\"kube-root-ca.crt\""
Jul 01 15:03:14.104231 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.104209    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9tvz/must-gather-l9j2j"]
Jul 01 15:03:14.271599 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.271570    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xvd\" (UniqueName: \"kubernetes.io/projected/fc79c19b-6584-446b-8ee5-6ba16e9afe52-kube-api-access-86xvd\") pod \"must-gather-l9j2j\" (UID: \"fc79c19b-6584-446b-8ee5-6ba16e9afe52\") " pod="openshift-must-gather-k9tvz/must-gather-l9j2j"
Jul 01 15:03:14.271768 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.271621    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fc79c19b-6584-446b-8ee5-6ba16e9afe52-must-gather-output\") pod \"must-gather-l9j2j\" (UID: \"fc79c19b-6584-446b-8ee5-6ba16e9afe52\") " pod="openshift-must-gather-k9tvz/must-gather-l9j2j"
Jul 01 15:03:14.372668 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.372593    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fc79c19b-6584-446b-8ee5-6ba16e9afe52-must-gather-output\") pod \"must-gather-l9j2j\" (UID: \"fc79c19b-6584-446b-8ee5-6ba16e9afe52\") " pod="openshift-must-gather-k9tvz/must-gather-l9j2j"
Jul 01 15:03:14.372820 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.372737    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xvd\" (UniqueName: \"kubernetes.io/projected/fc79c19b-6584-446b-8ee5-6ba16e9afe52-kube-api-access-86xvd\") pod \"must-gather-l9j2j\" (UID: \"fc79c19b-6584-446b-8ee5-6ba16e9afe52\") " pod="openshift-must-gather-k9tvz/must-gather-l9j2j"
Jul 01 15:03:14.372933 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.372916    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fc79c19b-6584-446b-8ee5-6ba16e9afe52-must-gather-output\") pod \"must-gather-l9j2j\" (UID: \"fc79c19b-6584-446b-8ee5-6ba16e9afe52\") " pod="openshift-must-gather-k9tvz/must-gather-l9j2j"
Jul 01 15:03:14.385355 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.385312    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xvd\" (UniqueName: \"kubernetes.io/projected/fc79c19b-6584-446b-8ee5-6ba16e9afe52-kube-api-access-86xvd\") pod \"must-gather-l9j2j\" (UID: \"fc79c19b-6584-446b-8ee5-6ba16e9afe52\") " pod="openshift-must-gather-k9tvz/must-gather-l9j2j"
Jul 01 15:03:14.401556 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.401531    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9tvz/must-gather-l9j2j"
Jul 01 15:03:14.528873 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.528847    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9tvz/must-gather-l9j2j"]
Jul 01 15:03:14.531196 ip-10-0-134-20 kubenswrapper[2558]: W0701 15:03:14.531170    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc79c19b_6584_446b_8ee5_6ba16e9afe52.slice/crio-2c788f4e0c59cb2a828a55b8786e3e6db071e87d1943ff741d7d7253ebe6f251 WatchSource:0}: Error finding container 2c788f4e0c59cb2a828a55b8786e3e6db071e87d1943ff741d7d7253ebe6f251: Status 404 returned error can't find the container with id 2c788f4e0c59cb2a828a55b8786e3e6db071e87d1943ff741d7d7253ebe6f251
Jul 01 15:03:14.616487 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:14.616454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9tvz/must-gather-l9j2j" event={"ID":"fc79c19b-6584-446b-8ee5-6ba16e9afe52","Type":"ContainerStarted","Data":"2c788f4e0c59cb2a828a55b8786e3e6db071e87d1943ff741d7d7253ebe6f251"}
Jul 01 15:03:16.629344 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:16.629296    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9tvz/must-gather-l9j2j" event={"ID":"fc79c19b-6584-446b-8ee5-6ba16e9afe52","Type":"ContainerStarted","Data":"28c6a93ee4faffa5f7958f5850a4eeb2357156f1cccfc365867c5fdb13a10421"}
Jul 01 15:03:16.629344 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:16.629351    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9tvz/must-gather-l9j2j" event={"ID":"fc79c19b-6584-446b-8ee5-6ba16e9afe52","Type":"ContainerStarted","Data":"fb4ac65aa1d418654762652b4e3f7af0ba76bb0bfbd1566256f20761b71626b6"}
Jul 01 15:03:16.649697 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:16.649636    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k9tvz/must-gather-l9j2j" podStartSLOduration=1.142532072 podStartE2EDuration="2.649613273s" totalImagesPullingTime="1.507081201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 15:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 15:03:16.647369274 +0000 UTC m=+1444.707948480" watchObservedRunningTime="2026-07-01 15:03:16.649613273 +0000 UTC m=+1444.710192465"
Jul 01 15:03:17.571015 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:17.570985    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t85xw_3cd6a69f-c352-40c4-a8b9-125145b3ec48/global-pull-secret-syncer/0.log"
Jul 01 15:03:17.753860 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:17.753826    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-phzkb_e0e28ac5-ab10-4108-b51f-185ea52ba4f2/konnectivity-agent/0.log"
Jul 01 15:03:17.804919 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:17.804897    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-20.ec2.internal_1bdd498e17258fb69bcdf5cab2dc7417/haproxy/0.log"
Jul 01 15:03:18.923070 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:18.923030    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-pp527/must-gather-hvfxm"]
Jul 01 15:03:18.924125 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:18.924092    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-pp527/must-gather-hvfxm" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" containerName="copy" containerID="cri-o://cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214" gracePeriod=2
Jul 01 15:03:18.928179 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:18.928145    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-pp527/must-gather-hvfxm"]
Jul 01 15:03:18.928664 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:03:18.928637    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hvfxm\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-pp527\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:03:19.316705 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.316577    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-pp527_must-gather-hvfxm_e8f8a310-4471-44da-be48-de71ce7a5798/copy/0.log"
Jul 01 15:03:19.318793 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.318517    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:03:19.321238 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:03:19.321201    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hvfxm\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-pp527\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:03:19.422052 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.422012    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8f8a310-4471-44da-be48-de71ce7a5798-must-gather-output\") pod \"e8f8a310-4471-44da-be48-de71ce7a5798\" (UID: \"e8f8a310-4471-44da-be48-de71ce7a5798\") "
Jul 01 15:03:19.422227 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.422089    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxbbc\" (UniqueName: \"kubernetes.io/projected/e8f8a310-4471-44da-be48-de71ce7a5798-kube-api-access-rxbbc\") pod \"e8f8a310-4471-44da-be48-de71ce7a5798\" (UID: \"e8f8a310-4471-44da-be48-de71ce7a5798\") "
Jul 01 15:03:19.423861 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.423830    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f8a310-4471-44da-be48-de71ce7a5798-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e8f8a310-4471-44da-be48-de71ce7a5798" (UID: "e8f8a310-4471-44da-be48-de71ce7a5798"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 15:03:19.432194 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.432149    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f8a310-4471-44da-be48-de71ce7a5798-kube-api-access-rxbbc" (OuterVolumeSpecName: "kube-api-access-rxbbc") pod "e8f8a310-4471-44da-be48-de71ce7a5798" (UID: "e8f8a310-4471-44da-be48-de71ce7a5798"). InnerVolumeSpecName "kube-api-access-rxbbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 15:03:19.523574 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.523529    2558 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8f8a310-4471-44da-be48-de71ce7a5798-must-gather-output\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:03:19.523574 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.523574    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxbbc\" (UniqueName: \"kubernetes.io/projected/e8f8a310-4471-44da-be48-de71ce7a5798-kube-api-access-rxbbc\") on node \"ip-10-0-134-20.ec2.internal\" DevicePath \"\""
Jul 01 15:03:19.645964 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.645936    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-pp527_must-gather-hvfxm_e8f8a310-4471-44da-be48-de71ce7a5798/copy/0.log"
Jul 01 15:03:19.650714 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.646616    2558 generic.go:356] "Generic (PLEG): container finished" podID="e8f8a310-4471-44da-be48-de71ce7a5798" containerID="cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214" exitCode=143
Jul 01 15:03:19.650714 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.646721    2558 scope.go:117] "RemoveContainer" containerID="cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214"
Jul 01 15:03:19.650714 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.646880    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:03:19.659602 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:03:19.659573    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hvfxm\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-pp527\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:03:19.666703 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:03:19.665261    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-hvfxm\" is forbidden: User \"system:node:ip-10-0-134-20.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-pp527\": no relationship found between node 'ip-10-0-134-20.ec2.internal' and this object" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" pod="openshift-must-gather-pp527/must-gather-hvfxm"
Jul 01 15:03:19.673514 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.673206    2558 scope.go:117] "RemoveContainer" containerID="a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4"
Jul 01 15:03:19.697092 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.696751    2558 scope.go:117] "RemoveContainer" containerID="cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214"
Jul 01 15:03:19.697511 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:03:19.697353    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214\": container with ID starting with cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214 not found: ID does not exist" containerID="cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214"
Jul 01 15:03:19.697511 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.697390    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214"} err="failed to get container status \"cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214\": rpc error: code = NotFound desc = could not find container \"cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214\": container with ID starting with cd4cfa5f0f94a53d873d00a3b3f465d59b37c5f8128eab5123450e37ccc26214 not found: ID does not exist"
Jul 01 15:03:19.697511 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.697416    2558 scope.go:117] "RemoveContainer" containerID="a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4"
Jul 01 15:03:19.697769 ip-10-0-134-20 kubenswrapper[2558]: E0701 15:03:19.697744    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4\": container with ID starting with a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4 not found: ID does not exist" containerID="a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4"
Jul 01 15:03:19.697839 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:19.697780    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4"} err="failed to get container status \"a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4\": rpc error: code = NotFound desc = could not find container \"a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4\": container with ID starting with a7e64cdb500f6f9454512f80ebce3cf0ced6cd2641c7496550150cd3dd5c66b4 not found: ID does not exist"
Jul 01 15:03:20.363119 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:20.363084    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" path="/var/lib/kubelet/pods/e8f8a310-4471-44da-be48-de71ce7a5798/volumes"
Jul 01 15:03:21.923760 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:21.923734    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_177046a9-1336-40b8-9861-bf9ed063ed47/alertmanager/0.log"
Jul 01 15:03:21.968259 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:21.968227    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_177046a9-1336-40b8-9861-bf9ed063ed47/config-reloader/0.log"
Jul 01 15:03:22.016622 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.016589    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_177046a9-1336-40b8-9861-bf9ed063ed47/kube-rbac-proxy-web/0.log"
Jul 01 15:03:22.070134 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.070102    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_177046a9-1336-40b8-9861-bf9ed063ed47/kube-rbac-proxy/0.log"
Jul 01 15:03:22.112942 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.112904    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_177046a9-1336-40b8-9861-bf9ed063ed47/kube-rbac-proxy-metric/0.log"
Jul 01 15:03:22.169278 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.169201    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_177046a9-1336-40b8-9861-bf9ed063ed47/prom-label-proxy/0.log"
Jul 01 15:03:22.225757 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.225718    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_177046a9-1336-40b8-9861-bf9ed063ed47/init-config-reloader/0.log"
Jul 01 15:03:22.321737 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.321666    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-4gdz2_bf0c1ee1-a21a-4358-832d-b627e5a823fe/kube-state-metrics/0.log"
Jul 01 15:03:22.367181 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.367149    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-4gdz2_bf0c1ee1-a21a-4358-832d-b627e5a823fe/kube-rbac-proxy-main/0.log"
Jul 01 15:03:22.414954 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.414923    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-4gdz2_bf0c1ee1-a21a-4358-832d-b627e5a823fe/kube-rbac-proxy-self/0.log"
Jul 01 15:03:22.458666 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.458566    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-85b4d78d6f-768rp_d892e1ea-9805-4c7a-bf8f-525cc3b52782/metrics-server/0.log"
Jul 01 15:03:22.511387 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.511359    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-cd8656b8-6kb5c_4c52135b-b9ab-416a-ba90-3f2082481551/monitoring-plugin/0.log"
Jul 01 15:03:22.561949 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.561916    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28696_6e3b7688-df75-43c7-91f1-be8eea205143/node-exporter/0.log"
Jul 01 15:03:22.603096 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.603014    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28696_6e3b7688-df75-43c7-91f1-be8eea205143/kube-rbac-proxy/0.log"
Jul 01 15:03:22.644384 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.644338    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28696_6e3b7688-df75-43c7-91f1-be8eea205143/init-textfile/0.log"
Jul 01 15:03:22.961776 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:22.961672    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-mrgzr_04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4/kube-rbac-proxy-main/0.log"
Jul 01 15:03:23.012038 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.011942    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-mrgzr_04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4/kube-rbac-proxy-self/0.log"
Jul 01 15:03:23.063652 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.063625    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-mrgzr_04cd3d1a-8124-458a-8a0a-cd6eb48e4ae4/openshift-state-metrics/0.log"
Jul 01 15:03:23.133236 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.133067    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0342580-1dea-4f27-8a3e-0e940b19bd5d/prometheus/0.log"
Jul 01 15:03:23.160587 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.160561    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0342580-1dea-4f27-8a3e-0e940b19bd5d/config-reloader/0.log"
Jul 01 15:03:23.197743 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.197403    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0342580-1dea-4f27-8a3e-0e940b19bd5d/thanos-sidecar/0.log"
Jul 01 15:03:23.229250 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.229209    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0342580-1dea-4f27-8a3e-0e940b19bd5d/kube-rbac-proxy-web/0.log"
Jul 01 15:03:23.256467 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.256423    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0342580-1dea-4f27-8a3e-0e940b19bd5d/kube-rbac-proxy/0.log"
Jul 01 15:03:23.285441 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.285408    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0342580-1dea-4f27-8a3e-0e940b19bd5d/kube-rbac-proxy-thanos/0.log"
Jul 01 15:03:23.315345 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.315321    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0342580-1dea-4f27-8a3e-0e940b19bd5d/init-config-reloader/0.log"
Jul 01 15:03:23.345777 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.345730    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-8jvxk_e0ac0443-bb91-4f99-b625-cc0871bd2723/prometheus-operator/0.log"
Jul 01 15:03:23.366138 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.366109    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-8jvxk_e0ac0443-bb91-4f99-b625-cc0871bd2723/kube-rbac-proxy/0.log"
Jul 01 15:03:23.432111 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.432029    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bff6f9c97-77rhh_cb67b5f7-7bdb-4799-83a7-c130d15841f0/telemeter-client/0.log"
Jul 01 15:03:23.461318 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.461290    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bff6f9c97-77rhh_cb67b5f7-7bdb-4799-83a7-c130d15841f0/reload/0.log"
Jul 01 15:03:23.487589 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.487560    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bff6f9c97-77rhh_cb67b5f7-7bdb-4799-83a7-c130d15841f0/kube-rbac-proxy/0.log"
Jul 01 15:03:23.522608 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.522572    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd8d4bdfd-4mn75_513d4970-c815-4490-a77d-c5b35ef74bc3/thanos-query/0.log"
Jul 01 15:03:23.558209 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.558179    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd8d4bdfd-4mn75_513d4970-c815-4490-a77d-c5b35ef74bc3/kube-rbac-proxy-web/0.log"
Jul 01 15:03:23.591253 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.591222    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd8d4bdfd-4mn75_513d4970-c815-4490-a77d-c5b35ef74bc3/kube-rbac-proxy/0.log"
Jul 01 15:03:23.620841 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.620813    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd8d4bdfd-4mn75_513d4970-c815-4490-a77d-c5b35ef74bc3/prom-label-proxy/0.log"
Jul 01 15:03:23.649000 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.648965    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd8d4bdfd-4mn75_513d4970-c815-4490-a77d-c5b35ef74bc3/kube-rbac-proxy-rules/0.log"
Jul 01 15:03:23.678669 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:23.678639    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd8d4bdfd-4mn75_513d4970-c815-4490-a77d-c5b35ef74bc3/kube-rbac-proxy-metrics/0.log"
Jul 01 15:03:25.560671 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.560635    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"]
Jul 01 15:03:25.561211 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.561165    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" containerName="gather"
Jul 01 15:03:25.561211 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.561182    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" containerName="gather"
Jul 01 15:03:25.561211 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.561193    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" containerName="copy"
Jul 01 15:03:25.561211 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.561201    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" containerName="copy"
Jul 01 15:03:25.561436 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.561296    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" containerName="gather"
Jul 01 15:03:25.561436 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.561308    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f8a310-4471-44da-be48-de71ce7a5798" containerName="copy"
Jul 01 15:03:25.565226 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.565190    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.576526 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.576486    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"]
Jul 01 15:03:25.584897 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.584866    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-podres\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.585053 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.584934    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-sys\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.585053 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.584994    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-proc\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.585195 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.585085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkjg\" (UniqueName: \"kubernetes.io/projected/38c1b762-d591-437b-b905-03cda5867616-kube-api-access-dlkjg\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.585195 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.585148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-lib-modules\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.686149 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.686113    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkjg\" (UniqueName: \"kubernetes.io/projected/38c1b762-d591-437b-b905-03cda5867616-kube-api-access-dlkjg\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.686325 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.686169    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-lib-modules\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.686325 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.686197    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-podres\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.686325 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.686231    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-sys\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.686325 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.686280    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-proc\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.686524 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.686359    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-podres\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.686524 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.686358    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-lib-modules\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.686524 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.686364    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-proc\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.686524 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.686362    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38c1b762-d591-437b-b905-03cda5867616-sys\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.696309 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.696276    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkjg\" (UniqueName: \"kubernetes.io/projected/38c1b762-d591-437b-b905-03cda5867616-kube-api-access-dlkjg\") pod \"perf-node-gather-daemonset-7bk5l\" (UID: \"38c1b762-d591-437b-b905-03cda5867616\") " pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:25.878931 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:25.878895    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:26.026244 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:26.025612    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"]
Jul 01 15:03:26.030021 ip-10-0-134-20 kubenswrapper[2558]: W0701 15:03:26.029963    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod38c1b762_d591_437b_b905_03cda5867616.slice/crio-f89159e12ba79c4541a68bcaf896d8aaf83046eacffacb15344148c9d797dbe6 WatchSource:0}: Error finding container f89159e12ba79c4541a68bcaf896d8aaf83046eacffacb15344148c9d797dbe6: Status 404 returned error can't find the container with id f89159e12ba79c4541a68bcaf896d8aaf83046eacffacb15344148c9d797dbe6
Jul 01 15:03:26.694417 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:26.694376    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l" event={"ID":"38c1b762-d591-437b-b905-03cda5867616","Type":"ContainerStarted","Data":"80c060f78ef253c021b243a1bb6d6c15899057cbd6972b14bd8bb20fa6bc7fa2"}
Jul 01 15:03:26.694931 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:26.694441    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l" event={"ID":"38c1b762-d591-437b-b905-03cda5867616","Type":"ContainerStarted","Data":"f89159e12ba79c4541a68bcaf896d8aaf83046eacffacb15344148c9d797dbe6"}
Jul 01 15:03:26.694931 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:26.694597    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:26.715367 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:26.715300    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l" podStartSLOduration=1.715280844 podStartE2EDuration="1.715280844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 15:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 15:03:26.712031111 +0000 UTC m=+1454.772610306" watchObservedRunningTime="2026-07-01 15:03:26.715280844 +0000 UTC m=+1454.775860037"
Jul 01 15:03:27.326526 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:27.326480    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l8l96_6ec001fb-e32a-4a0c-a80e-539e1f53f108/dns/0.log"
Jul 01 15:03:27.357007 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:27.356979    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l8l96_6ec001fb-e32a-4a0c-a80e-539e1f53f108/kube-rbac-proxy/0.log"
Jul 01 15:03:27.442898 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:27.442866    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4lrpd_7abab760-11aa-4e86-90df-31cbfb591873/dns-node-resolver/0.log"
Jul 01 15:03:28.096429 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:28.096403    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztcz2_0b05de0b-5d43-4566-b311-699a60bbd0cf/node-ca/0.log"
Jul 01 15:03:29.264547 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:29.264524    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9gwv6_881f795a-433f-43a3-8331-2b2ec04f2d83/serve-healthcheck-canary/0.log"
Jul 01 15:03:29.802585 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:29.802559    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vwk6n_13243029-0c8d-490c-a21b-437e7ca1b85d/kube-rbac-proxy/0.log"
Jul 01 15:03:29.828354 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:29.828320    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vwk6n_13243029-0c8d-490c-a21b-437e7ca1b85d/exporter/0.log"
Jul 01 15:03:29.856710 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:29.853859    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vwk6n_13243029-0c8d-490c-a21b-437e7ca1b85d/extractor/0.log"
Jul 01 15:03:32.080362 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:32.080330    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-9dcl5_f4feb8bc-17bd-437c-b9b1-b9f2fff655c5/manager/0.log"
Jul 01 15:03:32.234822 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:32.234794    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4msh6_6c402ac9-ae39-4840-b0fe-65794297382a/s3-init/0.log"
Jul 01 15:03:32.253282 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:32.253258    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4msh6_6c402ac9-ae39-4840-b0fe-65794297382a/download-hf-model/0.log"
Jul 01 15:03:32.709152 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:32.709127    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k9tvz/perf-node-gather-daemonset-7bk5l"
Jul 01 15:03:38.240908 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:38.240874    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5cqvx_9120aa3b-1ffb-484c-9471-601a7b69a1d5/kube-multus/0.log"
Jul 01 15:03:38.271745 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:38.271714    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwvqg_9abca8ef-87fb-4559-a1ec-f4fd48071abf/kube-multus-additional-cni-plugins/0.log"
Jul 01 15:03:38.329124 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:38.329080    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwvqg_9abca8ef-87fb-4559-a1ec-f4fd48071abf/egress-router-binary-copy/0.log"
Jul 01 15:03:38.361627 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:38.361604    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwvqg_9abca8ef-87fb-4559-a1ec-f4fd48071abf/cni-plugins/0.log"
Jul 01 15:03:38.388872 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:38.388852    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwvqg_9abca8ef-87fb-4559-a1ec-f4fd48071abf/bond-cni-plugin/0.log"
Jul 01 15:03:38.418186 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:38.418166    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwvqg_9abca8ef-87fb-4559-a1ec-f4fd48071abf/routeoverride-cni/0.log"
Jul 01 15:03:38.448707 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:38.448670    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwvqg_9abca8ef-87fb-4559-a1ec-f4fd48071abf/whereabouts-cni-bincopy/0.log"
Jul 01 15:03:38.475196 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:38.475172    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwvqg_9abca8ef-87fb-4559-a1ec-f4fd48071abf/whereabouts-cni/0.log"
Jul 01 15:03:39.162573 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:39.162546    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-khdj5_388279eb-7ee2-4130-9f63-3f54e10c7419/network-metrics-daemon/0.log"
Jul 01 15:03:39.186215 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:39.186190    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-khdj5_388279eb-7ee2-4130-9f63-3f54e10c7419/kube-rbac-proxy/0.log"
Jul 01 15:03:40.216704 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:40.216665    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-controller/0.log"
Jul 01 15:03:40.261895 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:40.261870    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/0.log"
Jul 01 15:03:40.261895 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:40.261898    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovn-acl-logging/1.log"
Jul 01 15:03:40.300809 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:40.300788    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/kube-rbac-proxy-node/0.log"
Jul 01 15:03:40.344680 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:40.344655    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 15:03:40.395171 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:40.395150    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/northd/0.log"
Jul 01 15:03:40.456063 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:40.456038    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/nbdb/0.log"
Jul 01 15:03:40.509034 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:40.508973    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/sbdb/0.log"
Jul 01 15:03:40.567773 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:40.567746    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtdlp_95dbc66a-5975-4106-ba5d-9564565d86f2/ovnkube-controller/0.log"
Jul 01 15:03:42.059914 ip-10-0-134-20 kubenswrapper[2558]: I0701 15:03:42.059828    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-llw6c_c5c4c5b8-1ba3-41f3-bee3-cf74ae04595c/network-check-target-container/0.log"