Sep 17 18:29:31.107860 ip-10-0-138-74 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:29:34.942189 ip-10-0-138-74 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:34.942189 ip-10-0-138-74 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:29:34.942189 ip-10-0-138-74 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:34.942189 ip-10-0-138-74 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:29:34.942189 ip-10-0-138-74 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:34.943025 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.942266    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:29:34.947354 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947336    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:34.947354 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947354    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947358    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947362    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947365    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947368    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947371    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947374    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947377    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947380    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947383    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947386    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947389    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947392    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947398    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947401    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947404    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947407    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947410    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947415    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:34.947426 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947419    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947423    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947426    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947429    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947434    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947438    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947442    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947445    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947447    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947450    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947453    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947456    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947460    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947462    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947465    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947468    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947470    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947473    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947476    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947478    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:34.947923 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947481    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947484    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947487    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947489    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947492    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947495    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947498    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947501    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947504    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947506    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947509    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947512    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947515    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947519    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947522    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947525    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947528    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947531    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947534    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:34.948445 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947536    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947539    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947542    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947545    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947547    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947550    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947566    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947569    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947572    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947576    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947578    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947581    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947584    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947587    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947591    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947594    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947597    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947600    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947603    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947606    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:34.948947 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947609    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947612    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947615    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947618    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947621    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947624    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947627    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947632    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947636    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947640    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947644    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947647    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947650    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947653    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947656    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947658    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947661    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947664    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947667    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.947670    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:34.949437 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949286    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949294    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949298    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949302    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949305    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949307    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949310    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949313    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949316    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949319    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949321    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949325    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949328    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949331    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949334    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949337    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949340    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949343    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949346    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:34.949979 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949349    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949354    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949358    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949362    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949365    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949368    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949371    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949374    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949377    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949379    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949382    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949385    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949388    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949391    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949393    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949396    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949399    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949402    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949405    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949407    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:34.950466 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949410    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949413    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949415    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949418    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949421    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949423    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949426    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949429    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949432    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949435    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949438    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949441    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949444    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949447    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949449    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949452    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949455    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949458    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949461    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949464    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:34.951003 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949467    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949469    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949472    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949475    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949479    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949481    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949485    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949488    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949491    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949493    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949496    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949499    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949501    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949504    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949507    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949509    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949512    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949515    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949520    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:34.951600 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949524    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949528    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949532    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949535    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949538    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949541    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949546    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949549    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949566    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949570    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949573    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949575    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949578    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949581    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949583    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949586    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949589    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949591    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949594    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949597    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:34.952095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949600    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949973    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949978    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949981    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949986    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949990    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949993    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949996    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.949999    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950002    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950005    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950007    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950010    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950014    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950016    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950020    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950022    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950025    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950028    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:34.952647 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950031    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950034    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950037    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950040    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950043    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950045    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950048    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950051    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950053    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950056    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950059    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950061    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950065    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950067    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950070    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950072    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950075    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950078    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950081    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:34.953159 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950084    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950087    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950090    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950093    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950096    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950098    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950102    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950105    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950112    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950115    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950118    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950120    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950124    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950127    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950129    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950132    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950135    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950138    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950141    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950143    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:34.953651 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950146    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950149    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950151    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950154    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950157    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950159    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950162    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950165    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950167    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950170    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950173    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950175    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950178    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950181    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950184    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950187    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950189    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950192    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950195    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950198    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:34.954141 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950201    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950204    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950207    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950210    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950214    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950217    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950220    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950223    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950225    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950228    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950230    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950233    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950236    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950239    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950241    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950246    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950249    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950252    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950255    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:34.954693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950258    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950261    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.950264    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950334    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950370    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950376    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950381    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950386    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950389    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950393    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950398    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950401    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950406    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950410    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950414    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950417    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950421    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950426    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950429    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950433    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950436    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950441    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950444    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:29:34.955168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950447    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950450    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950454    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950462    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950465    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950469    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950472    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950476    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950479    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950482    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950486    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950489    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950493    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950497    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950500    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950503    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950506    2561 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950509    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950513    2561 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950517    2561 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950520    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950525    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950528    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950532    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:29:34.955814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950535    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950539    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950543    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950546    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950549    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950566    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950569    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950573    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950576    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950579    2561 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950583    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950586    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950589    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950593    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950596    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950599    2561 flags.go:64] FLAG: --help="false"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950602    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-138-74.ec2.internal"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950605    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950608    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950612    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950615    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950619    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950622    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:29:34.956430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950625    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950628    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950631    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950634    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950638    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950641    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950645    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950648    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950651    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950654    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950657    2561 flags.go:64] FLAG: --lock-file=""
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950667    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950670    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950674    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950679    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950682    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950686    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950689    2561 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950691    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950695    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950698    2561 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950701    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950705    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950708    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950712    2561 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:29:34.957062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950716    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950719    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950722    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950725    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950728    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950731    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950734    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950741    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950744    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950747    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950750    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950753    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950759    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950762    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950767    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950771    2561 flags.go:64] FLAG: --port="10250"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950774    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950777    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-075f42493349e389d"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950780    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950790    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950793    2561 flags.go:64] FLAG: --register-node="true"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950796    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950800    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950803    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:29:34.957710 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950806    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950809    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950813    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950816    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950819    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950822    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950825    2561 flags.go:64] FLAG: --runonce="false"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950828    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950831    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950835    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950837    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950841    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950844    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950847    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950850    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950853    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950856    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950859    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950862    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950865    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950868    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950871    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950878    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950881    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950884    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:29:34.958728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950892    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950895    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950898    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950907    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950910    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950913    2561 flags.go:64] FLAG: --v="2"
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950917    2561 flags.go:64] FLAG: --version="false"
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950921    2561 flags.go:64] FLAG: --vmodule=""
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950925    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.950928    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951079    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951083    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951087    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951090    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951093    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951096    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951099    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951103    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951107    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951110    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951113    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:34.959876 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951116    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951119    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951122    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951125    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951127    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951140    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951144    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951147    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951153    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951155    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951158    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951161    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951164    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951166    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951197    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951207    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951211    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951214    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951217    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951220    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:34.960817 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951222    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951227    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951231    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951234    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951237    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951240    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951243    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951246    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951250    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951255    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951258    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951262    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951265    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951268    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951271    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951274    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951277    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951280    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951283    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:34.961693 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951285    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951290    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951293    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951296    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951299    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951301    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951304    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951307    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951310    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951320    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951324    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951328    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951331    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951334    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951336    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951339    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951342    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951345    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951348    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951351    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:34.962270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951354    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951357    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951361    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951364    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951367    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951370    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951373    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951375    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951378    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951381    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951384    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951386    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951389    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951393    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951396    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951399    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951402    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951404    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951407    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951411    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:34.962802 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951413    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951422    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951431    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951434    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951437    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951440    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951443    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951446    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.951449    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.951987    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.960392    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.960412    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.960440    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.960450    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:29:34.963308 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962077    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962100    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962106    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962113    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962126    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962143    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962147    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962260    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962265    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962268    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962272    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962276    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962279    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962282    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962285    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962288    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962291    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962294    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962299    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:34.963686 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962304    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962308    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962312    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962317    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962320    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962323    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962326    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962329    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962332    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962335    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962338    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962342    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962345    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962349    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962353    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962357    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962360    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962363    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962366    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962368    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:34.964201 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962371    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962374    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962377    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962380    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962383    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962386    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962389    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962392    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962395    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962397    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962400    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962403    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962407    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962410    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962412    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962416    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962418    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962421    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962424    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962428    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:34.964734 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962431    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962433    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962436    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962439    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962442    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962446    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962449    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962452    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962455    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962458    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962461    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962464    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962467    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962470    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962472    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962475    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962478    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962481    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962485    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:34.965237 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962488    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962490    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962493    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962496    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962499    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962502    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962505    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962508    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962510    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962513    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962516    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962519    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962523    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962526    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962529    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962533    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962536    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962539    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962542    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:34.965867 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962545    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962566    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.962574    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962707    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962712    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962715    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962718    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962721    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962724    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962727    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962730    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962732    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962735    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962738    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962741    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:34.966345 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962744    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962747    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962749    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962752    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962756    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962759    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962762    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962764    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962768    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962771    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962773    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962776    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962779    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962782    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962785    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962788    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962790    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962793    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962796    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962798    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:34.966755 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962801    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962804    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962807    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962810    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962814    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962819    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962823    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962826    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962829    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962832    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962835    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962838    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962841    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962844    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962847    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962849    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962852    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962855    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962858    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:34.967251 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962861    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962863    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962866    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962869    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962872    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962875    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962878    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962880    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962883    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962886    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962889    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962891    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962894    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962896    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962899    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962902    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962905    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962907    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962910    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962913    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:34.967735 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962915    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962918    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962922    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962926    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962929    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962932    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962935    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962938    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962940    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962943    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962946    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962949    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962952    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962954    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962957    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962960    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962962    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962965    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962968    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:34.968229 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962971    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962974    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962977    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962982    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962985    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962988    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962991    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962993    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:34.962996    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.963001    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.963109    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.965278    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.966222    2561 server.go:1019] "Starting client certificate rotation"
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.966323    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:34.968828 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.966354    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:34.969260 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.968938    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:34.970197 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.970168    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:34.974470 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.974454    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:29:34.976125 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.976110    2561 log.go:25] "Validated CRI v1 image API"
Sep 17 18:29:34.976953 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.976938    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:29:34.980813 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.980789    2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8b0f47b6-d486-47d5-9a25-a4df998e4c97:/dev/nvme0n1p4 a2ffa25a-0e23-401e-9e0e-bf20f887b4ca:/dev/nvme0n1p3]
Sep 17 18:29:34.980866 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.980813    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:29:34.985974 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.985760    2561 manager.go:216] Machine: {Timestamp:2026-09-17 18:29:34.985189282 +0000 UTC m=+2.264846303 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3120601 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2be6ec5d529c49bb159b4e096885eb SystemUUID:ec2be6ec-5d52-9c49-bb15-9b4e096885eb BootID:1f982dcc-80dc-473a-abb7-6f6502f791cf Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:98:72:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:98:72:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:40:6e:ba:ea:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:29:34.985974 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.985965    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:29:34.986105 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.986083    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:29:34.987767 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.987740    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:29:34.987901 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.987772    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:29:34.988989 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.988978    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:29:34.989030 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.988993    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:29:34.989030 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.989001    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:34.989030 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.989012    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:34.989116 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.989091    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:29:34.992187 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.992175    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:34.992313 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.992304    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:29:34.992346 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.992342    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:29:34.992377 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.992353    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:29:34.992377 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.992367    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:29:34.992377 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.992376    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:29:34.992470 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.992385    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:29:34.992750 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.992725    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:34.992750 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:34.992751    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:35.005451 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.005426    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:29:35.008077 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.008059    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:29:35.008152 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.008093    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:29:35.009234 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009217    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:29:35.009234 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009236    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009243    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009248    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009261    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009271    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009287    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009292    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009300    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009306    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009316    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009347    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:29:35.009371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009376    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:29:35.009922 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.009383    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:29:35.009922 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.009751    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:29:35.009922 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.009873    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:29:35.010022 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.010012    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:35.012204 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.012170    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-74.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:29:35.013252 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.013239    2561 server.go:1268] "Started kubelet"
Sep 17 18:29:35.013348 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.013308    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:29:35.013403 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.013338    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:29:35.013455 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.013367    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:29:35.014045 ip-10-0-138-74 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:29:35.014715 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.014429    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:29:35.014715 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.014616    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:29:35.018877 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.018858    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:29:35.021354 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.021337    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:29:35.021354 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.021348    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:35.021478 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.021410    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:35.022082 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022006    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:29:35.022082 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022059    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:29:35.022082 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022075    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:29:35.022259 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022244    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:29:35.022259 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022254    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:29:35.022492 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022453    2561 factory.go:153] Registering CRI-O factory
Sep 17 18:29:35.022492 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022469    2561 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:29:35.022643 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022542    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:29:35.022643 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022551    2561 factory.go:55] Registering systemd factory
Sep 17 18:29:35.022643 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022560    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:29:35.022643 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022576    2561 factory.go:103] Registering Raw factory
Sep 17 18:29:35.022643 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022589    2561 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:29:35.022994 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.022973    2561 manager.go:349] Starting recovery of all containers
Sep 17 18:29:35.031032 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.031012    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:29:35.049755 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.049726    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:29:35.050422 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.050394    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6cltf"
Sep 17 18:29:35.050576 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.050544    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6cltf"
Sep 17 18:29:35.051149 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.051126    2561 manager.go:354] Recovery completed
Sep 17 18:29:35.051388 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.051347    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:29:35.057753 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.057740    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:35.060293 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.060274    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:35.060370 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.060307    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:35.060370 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.060322    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:35.061462 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.061448    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:29:35.061462 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.061459    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:29:35.061600 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.061478    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:35.061600 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.061475    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:35.062290 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.062277    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:35.062337 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.062304    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:35.062337 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.062314    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:35.063176 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.063163    2561 policy_none.go:49] "None policy: Start"
Sep 17 18:29:35.063227 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.063180    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:29:35.063227 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.063190    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:29:35.064302 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.064290    2561 policy_none.go:47] "Start"
Sep 17 18:29:35.108383 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.108224    2561 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:29:35.171157 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.108416    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:29:35.171157 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.108428    2561 server.go:87] "Starting device plugin registration server"
Sep 17 18:29:35.171157 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.108658    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:29:35.171157 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.108677    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:29:35.171157 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.108800    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:29:35.171157 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.108893    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:29:35.171157 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.108900    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:29:35.171157 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.109363    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:29:35.171157 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.109410    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:35.210527 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.210453    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:35.212221 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.212201    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:29:35.212306 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.212254    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:29:35.212306 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.212275    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:29:35.212382 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.212314    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:29:35.212696 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.212664    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:35.212696 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.212696    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:35.212825 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.212711    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:35.212825 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.212741    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.215386 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.215363    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:35.219922 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.219907    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.219998 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.219928    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-74.ec2.internal\": node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:35.271101 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.271074    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:35.314928 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.314891    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"]
Sep 17 18:29:35.315022 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.314983    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:35.315965 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.315950    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:35.316034 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.315980    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:35.316034 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.315993    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:35.317641 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.317627    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:35.317773 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.317754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.317871 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.317791    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:35.318543 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.318526    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:35.318635 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.318551    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:35.318635 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.318575    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:35.318635 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.318532    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:35.318731 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.318637    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:35.318731 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.318650    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:35.320497 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.320482    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.320541 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.320512    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:35.321238 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.321222    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:35.321325 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.321252    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:35.321325 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.321267    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:35.323320 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.323295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/467b51f32347639980b9f38cde96784b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-74.ec2.internal\" (UID: \"467b51f32347639980b9f38cde96784b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.323418 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.323324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/757461f7e3fe40f81874c4962ad4ad49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"757461f7e3fe40f81874c4962ad4ad49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.323418 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.323352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/757461f7e3fe40f81874c4962ad4ad49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"757461f7e3fe40f81874c4962ad4ad49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.344465 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.344440    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-74.ec2.internal\" not found" node="ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.348197 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.348182    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-74.ec2.internal\" not found" node="ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.371135 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.371113    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:35.423926 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.423903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/757461f7e3fe40f81874c4962ad4ad49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"757461f7e3fe40f81874c4962ad4ad49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.424067 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.423931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/757461f7e3fe40f81874c4962ad4ad49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"757461f7e3fe40f81874c4962ad4ad49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.424067 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.423950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/467b51f32347639980b9f38cde96784b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-74.ec2.internal\" (UID: \"467b51f32347639980b9f38cde96784b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.424067 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.423978    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/467b51f32347639980b9f38cde96784b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-74.ec2.internal\" (UID: \"467b51f32347639980b9f38cde96784b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.424067 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.423998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/757461f7e3fe40f81874c4962ad4ad49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"757461f7e3fe40f81874c4962ad4ad49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.424067 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.424018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/757461f7e3fe40f81874c4962ad4ad49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"757461f7e3fe40f81874c4962ad4ad49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.471772 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.471698    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:35.575115 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.575073    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:35.649628 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.649600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.651862 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.651842    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:35.676158 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.676135    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:35.777639 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.777571    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:35.878092 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.878056    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:35.966531 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.966508    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:29:35.966918 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:35.966646    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:35.978790 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:35.978772    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found"
Sep 17 18:29:36.056629 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.056584    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:24:35 +0000 UTC" deadline="2028-06-03 19:21:02.628174988 +0000 UTC"
Sep 17 18:29:36.056629 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.056621    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15000h51m26.571556364s"
Sep 17 18:29:36.065355 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.065336    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:36.122035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.122008    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:36.142207 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.142186    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:36.143697 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.143681    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"
Sep 17 18:29:36.157129 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.157111    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:36.248472 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:36.248435    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757461f7e3fe40f81874c4962ad4ad49.slice/crio-d133a906806c396c713b1706b361fd93cecdf4855c82a8ffaeef5fd290229ced WatchSource:0}: Error finding container d133a906806c396c713b1706b361fd93cecdf4855c82a8ffaeef5fd290229ced: Status 404 returned error can't find the container with id d133a906806c396c713b1706b361fd93cecdf4855c82a8ffaeef5fd290229ced
Sep 17 18:29:36.248888 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:36.248846    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467b51f32347639980b9f38cde96784b.slice/crio-52cffe271244c0a209ab979fb2d163587e577e699fa3d312e49675ffe0fe3bca WatchSource:0}: Error finding container 52cffe271244c0a209ab979fb2d163587e577e699fa3d312e49675ffe0fe3bca: Status 404 returned error can't find the container with id 52cffe271244c0a209ab979fb2d163587e577e699fa3d312e49675ffe0fe3bca
Sep 17 18:29:36.254249 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.254235    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:29:36.352873 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.352806    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:36.486491 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.486450    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:36.993205 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:36.993175    2561 apiserver.go:52] "Watching apiserver"
Sep 17 18:29:37.006297 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.005288    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:29:37.006297 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.005820    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-w4s89","kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c","openshift-cluster-node-tuning-operator/tuned-l6754","openshift-multus/multus-additional-cni-plugins-c6sbr","openshift-network-diagnostics/network-check-target-szc74","openshift-ovn-kubernetes/ovnkube-node-bhfjv","kube-system/konnectivity-agent-4d7m6","openshift-dns/node-resolver-z6jng","openshift-image-registry/node-ca-xdtn6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal","openshift-multus/multus-8gvdb","openshift-multus/network-metrics-daemon-p5hr5","openshift-network-operator/iptables-alerter-zxrrd"]
Sep 17 18:29:37.008417 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.007676    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.009287 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.009264    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:37.009403 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.009351    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:37.010858 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.010467    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.010974 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.010952    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:29:37.011138 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.011121    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:29:37.011409 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.011389    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:29:37.011800 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.011781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.012089 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.012065    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:29:37.012317 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.012302    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:29:37.012317 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.012311    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:29:37.012814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.012637    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tmvt\""
Sep 17 18:29:37.012814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.012671    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:29:37.013900 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.013883    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pmtgv\""
Sep 17 18:29:37.014024 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.014010    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:29:37.014164 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.014152    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:29:37.014508 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.014475    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:37.014675 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.014660    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v6hp4\""
Sep 17 18:29:37.015255 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.014863    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.015255 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.015131    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:37.015384 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.015277    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:37.015384 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.015338    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:37.015902 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.015637    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.017122    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.017455    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.017653    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.017903    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.018255    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g766x\""
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.018414    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.018540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.018600    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.018610    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.018713    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.018749    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ss8rl\""
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.019009    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.020812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.020387    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.021692 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.021611    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:29:37.021919 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.021850    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:37.022039 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.022023    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:37.022852 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.022180    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:29:37.022852 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.022471    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s6thp\""
Sep 17 18:29:37.022852 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.022624    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:37.023030 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.022853    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:29:37.023158 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.023120    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bwktv\""
Sep 17 18:29:37.023606 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.023277    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.023606 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.023441    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:29:37.023606 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.023455    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:29:37.024870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.024716    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:29:37.025123 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.025072    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:29:37.025436 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.025283    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9shpq\""
Sep 17 18:29:37.025885 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.025866    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qnzd9\""
Sep 17 18:29:37.026096 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.026081    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:29:37.033800 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.033773    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3704c11-e37f-415e-b1dd-1b0facfeaa42-kubelet-config\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:37.033877 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.033813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:37.033962 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.033944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-systemd\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.034029 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.034011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-device-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.034160 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.034134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-log-socket\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.034793 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.034205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bs4j\" (UniqueName: \"kubernetes.io/projected/c5e409ea-38ae-4695-82a3-470500adce4d-kube-api-access-7bs4j\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.034857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.034814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-run-netns\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.034857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.034840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-hostroot\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.034934 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.034862    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-daemon-config\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.034934 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.034886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-run-multus-certs\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.034934 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.034909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.035050 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-run-k8s-cni-cncf-io\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.035099 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-var-lib-cni-multus\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.035099 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-etc-kubernetes\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.035198 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-host\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.035198 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035132    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/022b9b32-4714-42a9-b5f7-542385cc4352-iptables-alerter-script\") pod \"iptables-alerter-zxrrd\" (UID: \"022b9b32-4714-42a9-b5f7-542385cc4352\") " pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.035293 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbvj5\" (UniqueName: \"kubernetes.io/projected/1213f80e-4a5f-4930-ae2e-2734dad1d892-kube-api-access-kbvj5\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:37.035367 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-kubernetes\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.035405 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.035444 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-node-log\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.035444 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30074ef6-2972-4a1d-ad80-fa6f1319747e-ovn-node-metrics-cert\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.035548 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30074ef6-2972-4a1d-ad80-fa6f1319747e-ovnkube-script-lib\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.035548 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-cnibin\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.035548 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-var-lib-kubelet\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.036149 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.035552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-sysctl-conf\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.036414 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.036360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-run-netns\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9366a1c0-eed2-4f5b-9a11-ada95d186caf-konnectivity-ca\") pod \"konnectivity-agent-4d7m6\" (UID: \"9366a1c0-eed2-4f5b-9a11-ada95d186caf\") " pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-run-openvswitch\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-sysconfig\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038637    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-cnibin\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3704c11-e37f-415e-b1dd-1b0facfeaa42-dbus\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsfcp\" (UniqueName: \"kubernetes.io/projected/022b9b32-4714-42a9-b5f7-542385cc4352-kube-api-access-lsfcp\") pod \"iptables-alerter-zxrrd\" (UID: \"022b9b32-4714-42a9-b5f7-542385cc4352\") " pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-run-systemd\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-sysctl-d\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-lib-modules\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-system-cni-dir\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038910    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22c6c31c-3640-4498-bf6f-4023379af97a-serviceca\") pod \"node-ca-xdtn6\" (UID: \"22c6c31c-3640-4498-bf6f-4023379af97a\") " pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038745    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-kubelet\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.038996    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b2971f-a12f-494b-83aa-6c14a77418fd-tmp-dir\") pod \"node-resolver-z6jng\" (UID: \"97b2971f-a12f-494b-83aa-6c14a77418fd\") " pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.041172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-cni-bin\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22c6c31c-3640-4498-bf6f-4023379af97a-host\") pod \"node-ca-xdtn6\" (UID: \"22c6c31c-3640-4498-bf6f-4023379af97a\") " pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4pw4\" (UniqueName: \"kubernetes.io/projected/30074ef6-2972-4a1d-ad80-fa6f1319747e-kube-api-access-g4pw4\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-system-cni-dir\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-sys-fs\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-tuned\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039409    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9xt\" (UniqueName: \"kubernetes.io/projected/ab0ee06f-fab8-477c-a4a2-871e2d44308b-kube-api-access-5h9xt\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039464    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/022b9b32-4714-42a9-b5f7-542385cc4352-host-slash\") pod \"iptables-alerter-zxrrd\" (UID: \"022b9b32-4714-42a9-b5f7-542385cc4352\") " pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039549    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-slash\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-registration-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc62k\" (UniqueName: \"kubernetes.io/projected/97b2971f-a12f-494b-83aa-6c14a77418fd-kube-api-access-nc62k\") pod \"node-resolver-z6jng\" (UID: \"97b2971f-a12f-494b-83aa-6c14a77418fd\") " pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-var-lib-cni-bin\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlhrx\" (UniqueName: \"kubernetes.io/projected/22c6c31c-3640-4498-bf6f-4023379af97a-kube-api-access-rlhrx\") pod \"node-ca-xdtn6\" (UID: \"22c6c31c-3640-4498-bf6f-4023379af97a\") " pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9366a1c0-eed2-4f5b-9a11-ada95d186caf-agent-certs\") pod \"konnectivity-agent-4d7m6\" (UID: \"9366a1c0-eed2-4f5b-9a11-ada95d186caf\") " pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:29:37.042002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039759    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdvf\" (UniqueName: \"kubernetes.io/projected/7a825fbc-a453-4665-a358-cd2d1b343a3b-kube-api-access-mcdvf\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-sys\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-var-lib-kubelet\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039862    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039906    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-socket-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.039973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040159    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97b2971f-a12f-494b-83aa-6c14a77418fd-hosts-file\") pod \"node-resolver-z6jng\" (UID: \"97b2971f-a12f-494b-83aa-6c14a77418fd\") " pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab0ee06f-fab8-477c-a4a2-871e2d44308b-tmp\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-var-lib-openvswitch\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-cni-dir\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a825fbc-a453-4665-a358-cd2d1b343a3b-cni-binary-copy\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-socket-dir-parent\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-conf-dir\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-etc-openvswitch\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-run-ovn\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676p9\" (UniqueName: \"kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9\") pod \"network-check-target-szc74\" (UID: \"be355727-2ed7-493b-9067-832bffebf603\") " pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:37.042709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-modprobe-d\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.043308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-run\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.043308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040837    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:37.043308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-systemd-units\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.043308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.040900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-cni-netd\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.043308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.041036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30074ef6-2972-4a1d-ad80-fa6f1319747e-ovnkube-config\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.043308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.041231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30074ef6-2972-4a1d-ad80-fa6f1319747e-env-overrides\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.043308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.041275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwwv7\" (UniqueName: \"kubernetes.io/projected/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-kube-api-access-zwwv7\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.062565 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.062547    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cxvqf"
Sep 17 18:29:37.070578 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.070549    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cxvqf"
Sep 17 18:29:37.123693 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.123666    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:29:37.142251 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142210    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.142251 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4pw4\" (UniqueName: \"kubernetes.io/projected/30074ef6-2972-4a1d-ad80-fa6f1319747e-kube-api-access-g4pw4\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.142458 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-system-cni-dir\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.142458 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-sys-fs\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.142458 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-tuned\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.142458 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142323    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.142458 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9xt\" (UniqueName: \"kubernetes.io/projected/ab0ee06f-fab8-477c-a4a2-871e2d44308b-kube-api-access-5h9xt\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.142458 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.142458 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142392    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-sys-fs\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.142458 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-system-cni-dir\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.142458 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/022b9b32-4714-42a9-b5f7-542385cc4352-host-slash\") pod \"iptables-alerter-zxrrd\" (UID: \"022b9b32-4714-42a9-b5f7-542385cc4352\") " pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-slash\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/022b9b32-4714-42a9-b5f7-542385cc4352-host-slash\") pod \"iptables-alerter-zxrrd\" (UID: \"022b9b32-4714-42a9-b5f7-542385cc4352\") " pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-registration-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-slash\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc62k\" (UniqueName: \"kubernetes.io/projected/97b2971f-a12f-494b-83aa-6c14a77418fd-kube-api-access-nc62k\") pod \"node-resolver-z6jng\" (UID: \"97b2971f-a12f-494b-83aa-6c14a77418fd\") " pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-var-lib-cni-bin\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-registration-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlhrx\" (UniqueName: \"kubernetes.io/projected/22c6c31c-3640-4498-bf6f-4023379af97a-kube-api-access-rlhrx\") pod \"node-ca-xdtn6\" (UID: \"22c6c31c-3640-4498-bf6f-4023379af97a\") " pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-var-lib-cni-bin\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9366a1c0-eed2-4f5b-9a11-ada95d186caf-agent-certs\") pod \"konnectivity-agent-4d7m6\" (UID: \"9366a1c0-eed2-4f5b-9a11-ada95d186caf\") " pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdvf\" (UniqueName: \"kubernetes.io/projected/7a825fbc-a453-4665-a358-cd2d1b343a3b-kube-api-access-mcdvf\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-sys\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-var-lib-kubelet\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142737    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-sys\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-socket-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.142857 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142837    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-var-lib-kubelet\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97b2971f-a12f-494b-83aa-6c14a77418fd-hosts-file\") pod \"node-resolver-z6jng\" (UID: \"97b2971f-a12f-494b-83aa-6c14a77418fd\") " pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142885    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-socket-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142900    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab0ee06f-fab8-477c-a4a2-871e2d44308b-tmp\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-var-lib-openvswitch\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-cni-dir\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a825fbc-a453-4665-a358-cd2d1b343a3b-cni-binary-copy\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.142995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-socket-dir-parent\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143019    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-conf-dir\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-var-lib-openvswitch\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-etc-openvswitch\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-run-ovn\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676p9\" (UniqueName: \"kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9\") pod \"network-check-target-szc74\" (UID: \"be355727-2ed7-493b-9067-832bffebf603\") " pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97b2971f-a12f-494b-83aa-6c14a77418fd-hosts-file\") pod \"node-resolver-z6jng\" (UID: \"97b2971f-a12f-494b-83aa-6c14a77418fd\") " pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.143454 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-cni-dir\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-etc-openvswitch\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-conf-dir\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143380    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-socket-dir-parent\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-run-ovn\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-modprobe-d\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-run\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-systemd-units\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-cni-netd\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143638    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-modprobe-d\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30074ef6-2972-4a1d-ad80-fa6f1319747e-ovnkube-config\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30074ef6-2972-4a1d-ad80-fa6f1319747e-env-overrides\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.143709    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143726    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwwv7\" (UniqueName: \"kubernetes.io/projected/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-kube-api-access-zwwv7\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143753    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3704c11-e37f-415e-b1dd-1b0facfeaa42-kubelet-config\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:37.144146 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.143783    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret podName:e3704c11-e37f-415e-b1dd-1b0facfeaa42 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:37.643757233 +0000 UTC m=+4.923414239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret") pod "global-pull-secret-syncer-w4s89" (UID: "e3704c11-e37f-415e-b1dd-1b0facfeaa42") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3704c11-e37f-415e-b1dd-1b0facfeaa42-kubelet-config\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-systemd\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-device-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-systemd-units\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-log-socket\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bs4j\" (UniqueName: \"kubernetes.io/projected/c5e409ea-38ae-4695-82a3-470500adce4d-kube-api-access-7bs4j\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a825fbc-a453-4665-a358-cd2d1b343a3b-cni-binary-copy\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-run-netns\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-run-netns\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.143962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-hostroot\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-daemon-config\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-run\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-run-multus-certs\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-run-k8s-cni-cncf-io\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-var-lib-cni-multus\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.146687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-etc-kubernetes\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-host\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144167    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/022b9b32-4714-42a9-b5f7-542385cc4352-iptables-alerter-script\") pod \"iptables-alerter-zxrrd\" (UID: \"022b9b32-4714-42a9-b5f7-542385cc4352\") " pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbvj5\" (UniqueName: \"kubernetes.io/projected/1213f80e-4a5f-4930-ae2e-2734dad1d892-kube-api-access-kbvj5\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-kubernetes\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-node-log\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30074ef6-2972-4a1d-ad80-fa6f1319747e-ovnkube-config\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144292    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30074ef6-2972-4a1d-ad80-fa6f1319747e-ovn-node-metrics-cert\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.144297    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30074ef6-2972-4a1d-ad80-fa6f1319747e-ovnkube-script-lib\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.144341    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs podName:1213f80e-4a5f-4930-ae2e-2734dad1d892 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:37.644328004 +0000 UTC m=+4.923985027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs") pod "network-metrics-daemon-p5hr5" (UID: "1213f80e-4a5f-4930-ae2e-2734dad1d892") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144368    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-cnibin\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144400    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-var-lib-kubelet\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-sysctl-conf\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-run-netns\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9366a1c0-eed2-4f5b-9a11-ada95d186caf-konnectivity-ca\") pod \"konnectivity-agent-4d7m6\" (UID: \"9366a1c0-eed2-4f5b-9a11-ada95d186caf\") " pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:29:37.147177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-run-openvswitch\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-sysconfig\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144605    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-cnibin\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144659    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3704c11-e37f-415e-b1dd-1b0facfeaa42-dbus\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144687    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfcp\" (UniqueName: \"kubernetes.io/projected/022b9b32-4714-42a9-b5f7-542385cc4352-kube-api-access-lsfcp\") pod \"iptables-alerter-zxrrd\" (UID: \"022b9b32-4714-42a9-b5f7-542385cc4352\") " pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-run-systemd\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-sysctl-d\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-lib-modules\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-system-cni-dir\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-sysctl-conf\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22c6c31c-3640-4498-bf6f-4023379af97a-serviceca\") pod \"node-ca-xdtn6\" (UID: \"22c6c31c-3640-4498-bf6f-4023379af97a\") " pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-cnibin\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30074ef6-2972-4a1d-ad80-fa6f1319747e-env-overrides\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-kubelet\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144889    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-var-lib-kubelet\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-etc-kubernetes\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.147775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-hostroot\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b2971f-a12f-494b-83aa-6c14a77418fd-tmp-dir\") pod \"node-resolver-z6jng\" (UID: \"97b2971f-a12f-494b-83aa-6c14a77418fd\") " pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-run-netns\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-cni-netd\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-cni-bin\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-cni-bin\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22c6c31c-3640-4498-bf6f-4023379af97a-host\") pod \"node-ca-xdtn6\" (UID: \"22c6c31c-3640-4498-bf6f-4023379af97a\") " pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145117    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22c6c31c-3640-4498-bf6f-4023379af97a-host\") pod \"node-ca-xdtn6\" (UID: \"22c6c31c-3640-4498-bf6f-4023379af97a\") " pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-kubelet\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-run-k8s-cni-cncf-io\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-run-multus-certs\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145256    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b2971f-a12f-494b-83aa-6c14a77418fd-tmp-dir\") pod \"node-resolver-z6jng\" (UID: \"97b2971f-a12f-494b-83aa-6c14a77418fd\") " pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a825fbc-a453-4665-a358-cd2d1b343a3b-multus-daemon-config\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145324    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-systemd\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-log-socket\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145368    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5e409ea-38ae-4695-82a3-470500adce4d-device-dir\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.148342 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a825fbc-a453-4665-a358-cd2d1b343a3b-host-var-lib-cni-multus\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.144737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30074ef6-2972-4a1d-ad80-fa6f1319747e-ovnkube-script-lib\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-run-systemd\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145427    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-cnibin\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-sysctl-d\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-system-cni-dir\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9366a1c0-eed2-4f5b-9a11-ada95d186caf-konnectivity-ca\") pod \"konnectivity-agent-4d7m6\" (UID: \"9366a1c0-eed2-4f5b-9a11-ada95d186caf\") " pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-lib-modules\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3704c11-e37f-415e-b1dd-1b0facfeaa42-dbus\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145663    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-host\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-sysconfig\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.145845    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-run-openvswitch\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.146006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22c6c31c-3640-4498-bf6f-4023379af97a-serviceca\") pod \"node-ca-xdtn6\" (UID: \"22c6c31c-3640-4498-bf6f-4023379af97a\") " pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.146047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-kubernetes\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.146063    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30074ef6-2972-4a1d-ad80-fa6f1319747e-node-log\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.146151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.146385    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/022b9b32-4714-42a9-b5f7-542385cc4352-iptables-alerter-script\") pod \"iptables-alerter-zxrrd\" (UID: \"022b9b32-4714-42a9-b5f7-542385cc4352\") " pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.149056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.146555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab0ee06f-fab8-477c-a4a2-871e2d44308b-tmp\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.149833 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.146859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9366a1c0-eed2-4f5b-9a11-ada95d186caf-agent-certs\") pod \"konnectivity-agent-4d7m6\" (UID: \"9366a1c0-eed2-4f5b-9a11-ada95d186caf\") " pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:29:37.149833 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.148035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30074ef6-2972-4a1d-ad80-fa6f1319747e-ovn-node-metrics-cert\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.151570 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.151550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab0ee06f-fab8-477c-a4a2-871e2d44308b-etc-tuned\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.155286 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.155169    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:37.155286 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.155199    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:37.155286 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.155212    2561 projected.go:196] Error preparing data for projected volume kube-api-access-676p9 for pod openshift-network-diagnostics/network-check-target-szc74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:37.155286 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.155265    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9 podName:be355727-2ed7-493b-9067-832bffebf603 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:37.655250348 +0000 UTC m=+4.934907355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-676p9" (UniqueName: "kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9") pod "network-check-target-szc74" (UID: "be355727-2ed7-493b-9067-832bffebf603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:37.157561 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.157540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfcp\" (UniqueName: \"kubernetes.io/projected/022b9b32-4714-42a9-b5f7-542385cc4352-kube-api-access-lsfcp\") pod \"iptables-alerter-zxrrd\" (UID: \"022b9b32-4714-42a9-b5f7-542385cc4352\") " pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.158030 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.158012    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc62k\" (UniqueName: \"kubernetes.io/projected/97b2971f-a12f-494b-83aa-6c14a77418fd-kube-api-access-nc62k\") pod \"node-resolver-z6jng\" (UID: \"97b2971f-a12f-494b-83aa-6c14a77418fd\") " pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.162592 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.162091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwwv7\" (UniqueName: \"kubernetes.io/projected/cdb005e3-9caa-4fd7-ae41-bc8936506eb2-kube-api-access-zwwv7\") pod \"multus-additional-cni-plugins-c6sbr\" (UID: \"cdb005e3-9caa-4fd7-ae41-bc8936506eb2\") " pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.165011 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.164971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4pw4\" (UniqueName: \"kubernetes.io/projected/30074ef6-2972-4a1d-ad80-fa6f1319747e-kube-api-access-g4pw4\") pod \"ovnkube-node-bhfjv\" (UID: \"30074ef6-2972-4a1d-ad80-fa6f1319747e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.165452 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.165409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlhrx\" (UniqueName: \"kubernetes.io/projected/22c6c31c-3640-4498-bf6f-4023379af97a-kube-api-access-rlhrx\") pod \"node-ca-xdtn6\" (UID: \"22c6c31c-3640-4498-bf6f-4023379af97a\") " pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.166737 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.166716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbvj5\" (UniqueName: \"kubernetes.io/projected/1213f80e-4a5f-4930-ae2e-2734dad1d892-kube-api-access-kbvj5\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:37.166820 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.166716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bs4j\" (UniqueName: \"kubernetes.io/projected/c5e409ea-38ae-4695-82a3-470500adce4d-kube-api-access-7bs4j\") pod \"aws-ebs-csi-driver-node-2bw4c\" (UID: \"c5e409ea-38ae-4695-82a3-470500adce4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.168558 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.168522    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9xt\" (UniqueName: \"kubernetes.io/projected/ab0ee06f-fab8-477c-a4a2-871e2d44308b-kube-api-access-5h9xt\") pod \"tuned-l6754\" (UID: \"ab0ee06f-fab8-477c-a4a2-871e2d44308b\") " pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.168687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.168671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdvf\" (UniqueName: \"kubernetes.io/projected/7a825fbc-a453-4665-a358-cd2d1b343a3b-kube-api-access-mcdvf\") pod \"multus-8gvdb\" (UID: \"7a825fbc-a453-4665-a358-cd2d1b343a3b\") " pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.216555 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.216461    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" event={"ID":"757461f7e3fe40f81874c4962ad4ad49","Type":"ContainerStarted","Data":"d133a906806c396c713b1706b361fd93cecdf4855c82a8ffaeef5fd290229ced"}
Sep 17 18:29:37.217672 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.217648    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" event={"ID":"467b51f32347639980b9f38cde96784b","Type":"ContainerStarted","Data":"52cffe271244c0a209ab979fb2d163587e577e699fa3d312e49675ffe0fe3bca"}
Sep 17 18:29:37.326469 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.326437    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:29:37.334456 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.334430    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c"
Sep 17 18:29:37.348647 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.348626    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l6754"
Sep 17 18:29:37.355615 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.355593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6sbr"
Sep 17 18:29:37.364723 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.364704    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z6jng"
Sep 17 18:29:37.371523 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.371505    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zxrrd"
Sep 17 18:29:37.378445 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.378421    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:37.384573 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.384554    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xdtn6"
Sep 17 18:29:37.392333 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.392315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:29:37.397985 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.397967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8gvdb"
Sep 17 18:29:37.648704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.648618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:37.648704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.648670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:37.648923 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.648792    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:37.648923 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.648829    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:37.648923 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.648866    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret podName:e3704c11-e37f-415e-b1dd-1b0facfeaa42 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:38.648847311 +0000 UTC m=+5.928504323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret") pod "global-pull-secret-syncer-w4s89" (UID: "e3704c11-e37f-415e-b1dd-1b0facfeaa42") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:37.648923 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.648887    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs podName:1213f80e-4a5f-4930-ae2e-2734dad1d892 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:38.648875967 +0000 UTC m=+5.928532974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs") pod "network-metrics-daemon-p5hr5" (UID: "1213f80e-4a5f-4930-ae2e-2734dad1d892") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:37.749731 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:37.749694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676p9\" (UniqueName: \"kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9\") pod \"network-check-target-szc74\" (UID: \"be355727-2ed7-493b-9067-832bffebf603\") " pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:37.749902 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.749884    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:37.749973 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.749940    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:37.749973 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.749954    2561 projected.go:196] Error preparing data for projected volume kube-api-access-676p9 for pod openshift-network-diagnostics/network-check-target-szc74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:37.750066 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:37.750019    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9 podName:be355727-2ed7-493b-9067-832bffebf603 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:38.750000823 +0000 UTC m=+6.029657844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-676p9" (UniqueName: "kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9") pod "network-check-target-szc74" (UID: "be355727-2ed7-493b-9067-832bffebf603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:38.072462 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.072415    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:37 +0000 UTC" deadline="2028-04-11 06:34:51.886385161 +0000 UTC"
Sep 17 18:29:38.072462 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.072458    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13716h5m13.813930525s"
Sep 17 18:29:38.109847 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:38.109819    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb005e3_9caa_4fd7_ae41_bc8936506eb2.slice/crio-55bdacf7ed20db759597ce48bfaae1aff0a7df7ad8ebef27e49720eb64c94509 WatchSource:0}: Error finding container 55bdacf7ed20db759597ce48bfaae1aff0a7df7ad8ebef27e49720eb64c94509: Status 404 returned error can't find the container with id 55bdacf7ed20db759597ce48bfaae1aff0a7df7ad8ebef27e49720eb64c94509
Sep 17 18:29:38.111973 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:38.111908    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a825fbc_a453_4665_a358_cd2d1b343a3b.slice/crio-faa02be874bdb0851cffbc87901b0e87e4a73b88bd899f2049135cdf76f9dc3c WatchSource:0}: Error finding container faa02be874bdb0851cffbc87901b0e87e4a73b88bd899f2049135cdf76f9dc3c: Status 404 returned error can't find the container with id faa02be874bdb0851cffbc87901b0e87e4a73b88bd899f2049135cdf76f9dc3c
Sep 17 18:29:38.115810 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:38.115617    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e409ea_38ae_4695_82a3_470500adce4d.slice/crio-b93b6e71546b85ddd3125b6452090e43bc5e78c15e0985c56f0aefebefbc5107 WatchSource:0}: Error finding container b93b6e71546b85ddd3125b6452090e43bc5e78c15e0985c56f0aefebefbc5107: Status 404 returned error can't find the container with id b93b6e71546b85ddd3125b6452090e43bc5e78c15e0985c56f0aefebefbc5107
Sep 17 18:29:38.117847 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:38.117819    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9366a1c0_eed2_4f5b_9a11_ada95d186caf.slice/crio-1166ada2d36075895ddfc6c1a1d9e5f86bcc3876b65cce32ff263faa643eeda5 WatchSource:0}: Error finding container 1166ada2d36075895ddfc6c1a1d9e5f86bcc3876b65cce32ff263faa643eeda5: Status 404 returned error can't find the container with id 1166ada2d36075895ddfc6c1a1d9e5f86bcc3876b65cce32ff263faa643eeda5
Sep 17 18:29:38.119583 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:38.119005    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022b9b32_4714_42a9_b5f7_542385cc4352.slice/crio-a688c5ff64732796d5bede88a16e6e82007f1a2c3ebf4c238811e5fab687cc53 WatchSource:0}: Error finding container a688c5ff64732796d5bede88a16e6e82007f1a2c3ebf4c238811e5fab687cc53: Status 404 returned error can't find the container with id a688c5ff64732796d5bede88a16e6e82007f1a2c3ebf4c238811e5fab687cc53
Sep 17 18:29:38.119583 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:38.119513    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30074ef6_2972_4a1d_ad80_fa6f1319747e.slice/crio-734c06c9d1b919f4f071bedb6899755fbc6f630118d88b0b5d80ad3979b9bb0a WatchSource:0}: Error finding container 734c06c9d1b919f4f071bedb6899755fbc6f630118d88b0b5d80ad3979b9bb0a: Status 404 returned error can't find the container with id 734c06c9d1b919f4f071bedb6899755fbc6f630118d88b0b5d80ad3979b9bb0a
Sep 17 18:29:38.122275 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:38.122250    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c6c31c_3640_4498_bf6f_4023379af97a.slice/crio-2d1ec2257371b6bec1b428a675c713024beb212e7d32544e039a5d83d3f229f3 WatchSource:0}: Error finding container 2d1ec2257371b6bec1b428a675c713024beb212e7d32544e039a5d83d3f229f3: Status 404 returned error can't find the container with id 2d1ec2257371b6bec1b428a675c713024beb212e7d32544e039a5d83d3f229f3
Sep 17 18:29:38.123578 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:29:38.123558    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0ee06f_fab8_477c_a4a2_871e2d44308b.slice/crio-96ac0e28efe240cb4976b0b4fada1759dac114a5f65b2e10ba504700d726c866 WatchSource:0}: Error finding container 96ac0e28efe240cb4976b0b4fada1759dac114a5f65b2e10ba504700d726c866: Status 404 returned error can't find the container with id 96ac0e28efe240cb4976b0b4fada1759dac114a5f65b2e10ba504700d726c866
Sep 17 18:29:38.224428 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.224403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zxrrd" event={"ID":"022b9b32-4714-42a9-b5f7-542385cc4352","Type":"ContainerStarted","Data":"a688c5ff64732796d5bede88a16e6e82007f1a2c3ebf4c238811e5fab687cc53"}
Sep 17 18:29:38.226337 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.226307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sbr" event={"ID":"cdb005e3-9caa-4fd7-ae41-bc8936506eb2","Type":"ContainerStarted","Data":"55bdacf7ed20db759597ce48bfaae1aff0a7df7ad8ebef27e49720eb64c94509"}
Sep 17 18:29:38.232023 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.231996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" event={"ID":"467b51f32347639980b9f38cde96784b","Type":"ContainerStarted","Data":"0f8da1a7a6710c5a3ea6d6285780e42997d8baf80dd050db81c0ce182b96338f"}
Sep 17 18:29:38.233202 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.233182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" event={"ID":"30074ef6-2972-4a1d-ad80-fa6f1319747e","Type":"ContainerStarted","Data":"734c06c9d1b919f4f071bedb6899755fbc6f630118d88b0b5d80ad3979b9bb0a"}
Sep 17 18:29:38.235277 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.235256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4d7m6" event={"ID":"9366a1c0-eed2-4f5b-9a11-ada95d186caf","Type":"ContainerStarted","Data":"1166ada2d36075895ddfc6c1a1d9e5f86bcc3876b65cce32ff263faa643eeda5"}
Sep 17 18:29:38.236396 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.236374    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z6jng" event={"ID":"97b2971f-a12f-494b-83aa-6c14a77418fd","Type":"ContainerStarted","Data":"958894bfc6642b0f9d89add21a32a99a24b55e1fb72c4deee467d5ca18a05360"}
Sep 17 18:29:38.238409 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.238389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c" event={"ID":"c5e409ea-38ae-4695-82a3-470500adce4d","Type":"ContainerStarted","Data":"b93b6e71546b85ddd3125b6452090e43bc5e78c15e0985c56f0aefebefbc5107"}
Sep 17 18:29:38.241658 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.241637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8gvdb" event={"ID":"7a825fbc-a453-4665-a358-cd2d1b343a3b","Type":"ContainerStarted","Data":"faa02be874bdb0851cffbc87901b0e87e4a73b88bd899f2049135cdf76f9dc3c"}
Sep 17 18:29:38.243121 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.243100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l6754" event={"ID":"ab0ee06f-fab8-477c-a4a2-871e2d44308b","Type":"ContainerStarted","Data":"96ac0e28efe240cb4976b0b4fada1759dac114a5f65b2e10ba504700d726c866"}
Sep 17 18:29:38.245089 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.245067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xdtn6" event={"ID":"22c6c31c-3640-4498-bf6f-4023379af97a","Type":"ContainerStarted","Data":"2d1ec2257371b6bec1b428a675c713024beb212e7d32544e039a5d83d3f229f3"}
Sep 17 18:29:38.251324 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.251287    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" podStartSLOduration=2.251276711 podStartE2EDuration="2.251276711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:38.251090141 +0000 UTC m=+5.530747171" watchObservedRunningTime="2026-09-17 18:29:38.251276711 +0000 UTC m=+5.530933750"
Sep 17 18:29:38.655240 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.654924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:38.655240 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.655198    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:38.655456 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:38.655363    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:38.655456 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:38.655429    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs podName:1213f80e-4a5f-4930-ae2e-2734dad1d892 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:40.655409917 +0000 UTC m=+7.935066926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs") pod "network-metrics-daemon-p5hr5" (UID: "1213f80e-4a5f-4930-ae2e-2734dad1d892") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:38.655941 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:38.655896    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:38.656036 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:38.655957    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret podName:e3704c11-e37f-415e-b1dd-1b0facfeaa42 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:40.65594092 +0000 UTC m=+7.935597943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret") pod "global-pull-secret-syncer-w4s89" (UID: "e3704c11-e37f-415e-b1dd-1b0facfeaa42") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:38.756076 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:38.756043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676p9\" (UniqueName: \"kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9\") pod \"network-check-target-szc74\" (UID: \"be355727-2ed7-493b-9067-832bffebf603\") " pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:38.756257 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:38.756239    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:38.756317 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:38.756263    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:38.756317 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:38.756275    2561 projected.go:196] Error preparing data for projected volume kube-api-access-676p9 for pod openshift-network-diagnostics/network-check-target-szc74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:38.756417 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:38.756335    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9 podName:be355727-2ed7-493b-9067-832bffebf603 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:40.75631663 +0000 UTC m=+8.035973659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-676p9" (UniqueName: "kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9") pod "network-check-target-szc74" (UID: "be355727-2ed7-493b-9067-832bffebf603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:39.073590 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:39.073503    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:37 +0000 UTC" deadline="2028-05-07 08:29:49.121384703 +0000 UTC"
Sep 17 18:29:39.073590 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:39.073562    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14342h0m10.047827225s"
Sep 17 18:29:39.213666 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:39.213630    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:39.213837 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:39.213789    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:39.213837 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:39.213821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:39.213966 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:39.213876    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:39.214016 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:39.213968    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:39.214067 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:39.214044    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:39.263652 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:39.263611    2561 generic.go:356] "Generic (PLEG): container finished" podID="757461f7e3fe40f81874c4962ad4ad49" containerID="aca2afea6cf4a9cd674748a1136fb9b4da82311caaed3ef071dfa98852ca1263" exitCode=0
Sep 17 18:29:39.264755 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:39.264728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" event={"ID":"757461f7e3fe40f81874c4962ad4ad49","Type":"ContainerDied","Data":"aca2afea6cf4a9cd674748a1136fb9b4da82311caaed3ef071dfa98852ca1263"}
Sep 17 18:29:40.275764 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:40.275723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" event={"ID":"757461f7e3fe40f81874c4962ad4ad49","Type":"ContainerStarted","Data":"92215a685ef9f2d1baf606789fc73b8c932b7971091897d12008da04aeee1df8"}
Sep 17 18:29:40.672232 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:40.672201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:40.672232 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:40.672243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:40.672462 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:40.672342    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:40.672462 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:40.672386    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs podName:1213f80e-4a5f-4930-ae2e-2734dad1d892 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.672372855 +0000 UTC m=+11.952029862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs") pod "network-metrics-daemon-p5hr5" (UID: "1213f80e-4a5f-4930-ae2e-2734dad1d892") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:40.672634 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:40.672606    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:40.672686 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:40.672673    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret podName:e3704c11-e37f-415e-b1dd-1b0facfeaa42 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.672657902 +0000 UTC m=+11.952314926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret") pod "global-pull-secret-syncer-w4s89" (UID: "e3704c11-e37f-415e-b1dd-1b0facfeaa42") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:40.773147 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:40.773099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676p9\" (UniqueName: \"kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9\") pod \"network-check-target-szc74\" (UID: \"be355727-2ed7-493b-9067-832bffebf603\") " pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:40.773389 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:40.773363    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:40.773389 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:40.773389    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:40.773551 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:40.773402    2561 projected.go:196] Error preparing data for projected volume kube-api-access-676p9 for pod openshift-network-diagnostics/network-check-target-szc74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:40.773551 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:40.773455    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9 podName:be355727-2ed7-493b-9067-832bffebf603 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.773437551 +0000 UTC m=+12.053094564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-676p9" (UniqueName: "kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9") pod "network-check-target-szc74" (UID: "be355727-2ed7-493b-9067-832bffebf603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:41.212637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:41.212584    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:41.212813 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:41.212751    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:41.213182 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:41.213158    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:41.213279 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:41.213257    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:41.213618 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:41.213415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:41.213618 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:41.213537    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:43.213012 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:43.212981    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:43.213350 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:43.213162    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:43.213510 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:43.213471    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:43.213613 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:43.213593    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:43.213730 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:43.213717    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:43.213814 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:43.213800    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:44.710852 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:44.710810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:44.710852 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:44.710854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:44.711440 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:44.711016    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:44.711440 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:44.711091    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs podName:1213f80e-4a5f-4930-ae2e-2734dad1d892 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:52.711070107 +0000 UTC m=+19.990727136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs") pod "network-metrics-daemon-p5hr5" (UID: "1213f80e-4a5f-4930-ae2e-2734dad1d892") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:44.711440 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:44.711108    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:44.711440 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:44.711168    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret podName:e3704c11-e37f-415e-b1dd-1b0facfeaa42 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:52.71114904 +0000 UTC m=+19.990806058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret") pod "global-pull-secret-syncer-w4s89" (UID: "e3704c11-e37f-415e-b1dd-1b0facfeaa42") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:44.812196 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:44.812150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676p9\" (UniqueName: \"kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9\") pod \"network-check-target-szc74\" (UID: \"be355727-2ed7-493b-9067-832bffebf603\") " pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:44.812365 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:44.812343    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:44.812425 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:44.812367    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:44.812425 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:44.812414    2561 projected.go:196] Error preparing data for projected volume kube-api-access-676p9 for pod openshift-network-diagnostics/network-check-target-szc74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:44.812577 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:44.812496    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9 podName:be355727-2ed7-493b-9067-832bffebf603 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:52.812459889 +0000 UTC m=+20.092116910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-676p9" (UniqueName: "kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9") pod "network-check-target-szc74" (UID: "be355727-2ed7-493b-9067-832bffebf603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:45.219198 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:45.216311    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:45.219198 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:45.216498    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:45.219198 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:45.217072    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:45.219198 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:45.217208    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:45.219198 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:45.217324    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:45.219198 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:45.217373    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:47.213415 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:47.213382    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:47.213859 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:47.213384    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:47.213859 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:47.213518    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:47.213859 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:47.213610    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:47.213859 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:47.213391    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:47.213859 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:47.213709    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:49.219532 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:49.216887    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:49.219532 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:49.216923    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:49.219532 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:49.216623    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:49.219532 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:49.217064    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:49.219532 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:49.217203    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:49.219532 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:49.217707    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:51.213384 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:51.213351    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:51.213384 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:51.213374    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:51.213937 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:51.213355    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:51.213937 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:51.213502    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:51.213937 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:51.213585    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:51.213937 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:51.213693    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:52.769522 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:52.769468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:52.769940 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:52.769532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:52.769940 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:52.769618    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:52.769940 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:52.769646    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:52.769940 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:52.769696    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret podName:e3704c11-e37f-415e-b1dd-1b0facfeaa42 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.769678278 +0000 UTC m=+36.049335286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret") pod "global-pull-secret-syncer-w4s89" (UID: "e3704c11-e37f-415e-b1dd-1b0facfeaa42") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:52.769940 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:52.769713    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs podName:1213f80e-4a5f-4930-ae2e-2734dad1d892 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.769705476 +0000 UTC m=+36.049362483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs") pod "network-metrics-daemon-p5hr5" (UID: "1213f80e-4a5f-4930-ae2e-2734dad1d892") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:52.870172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:52.870136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676p9\" (UniqueName: \"kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9\") pod \"network-check-target-szc74\" (UID: \"be355727-2ed7-493b-9067-832bffebf603\") " pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:52.870335 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:52.870299    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:52.870335 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:52.870318    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:52.870335 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:52.870328    2561 projected.go:196] Error preparing data for projected volume kube-api-access-676p9 for pod openshift-network-diagnostics/network-check-target-szc74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:52.870454 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:52.870383    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9 podName:be355727-2ed7-493b-9067-832bffebf603 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.870366383 +0000 UTC m=+36.150023413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-676p9" (UniqueName: "kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9") pod "network-check-target-szc74" (UID: "be355727-2ed7-493b-9067-832bffebf603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:53.212860 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:53.212827    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:53.212860 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:53.212854    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:53.213099 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:53.212827    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:53.213099 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:53.212962    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:53.213099 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:53.213025    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:53.213243 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:53.213104    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:55.214647 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:55.214249    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:55.214647 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:55.214345    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:55.214647 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:55.214357    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:55.214647 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:55.214391    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:55.214647 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:55.214512    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:55.214647 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:55.214575    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:55.314812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:55.314697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4d7m6" event={"ID":"9366a1c0-eed2-4f5b-9a11-ada95d186caf","Type":"ContainerStarted","Data":"e058c8017effc1d71f96998e36e935d0ecbb6ccfd27081b6908354255ec46083"}
Sep 17 18:29:55.330886 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:55.330842    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" podStartSLOduration=19.330822449 podStartE2EDuration="19.330822449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:40.292789868 +0000 UTC m=+7.572446898" watchObservedRunningTime="2026-09-17 18:29:55.330822449 +0000 UTC m=+22.610479479"
Sep 17 18:29:55.331453 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:55.331424    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4d7m6" podStartSLOduration=10.568050082 podStartE2EDuration="20.331414383s" totalImagesPullingTime="9.763364301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:55.330814756 +0000 UTC m=+22.610471784" watchObservedRunningTime="2026-09-17 18:29:55.331414383 +0000 UTC m=+22.611071468"
Sep 17 18:29:55.482606 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:55.482404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:56.325475 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.325446    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:29:56.326238 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.325806    2561 generic.go:356] "Generic (PLEG): container finished" podID="30074ef6-2972-4a1d-ad80-fa6f1319747e" containerID="e01b5c3685da798fa4c1bac89f8fffde99c09b8758a8cbfbf876dcdcadf488b4" exitCode=1
Sep 17 18:29:56.326238 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.325888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" event={"ID":"30074ef6-2972-4a1d-ad80-fa6f1319747e","Type":"ContainerStarted","Data":"2fde56cae9e3781dff4e8952aa484c2f2f233f28badb9c4ac11b9f183c079aac"}
Sep 17 18:29:56.326238 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.325984    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" event={"ID":"30074ef6-2972-4a1d-ad80-fa6f1319747e","Type":"ContainerStarted","Data":"c7b5fd724bb5b07b96fc6dc8070e265cc02d0365fc717764833bbbbeeda4f9d9"}
Sep 17 18:29:56.326238 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.326016    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" event={"ID":"30074ef6-2972-4a1d-ad80-fa6f1319747e","Type":"ContainerStarted","Data":"171d0af5e07e78294f6cd73ad41c8f2f85977b23224a95e7a5dc93b1c9cb3d8b"}
Sep 17 18:29:56.326238 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.326044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" event={"ID":"30074ef6-2972-4a1d-ad80-fa6f1319747e","Type":"ContainerStarted","Data":"02b0ad3620bfe22135ac77bb8c05bc277d8631cd96c0559386808f332825679b"}
Sep 17 18:29:56.326238 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.326079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" event={"ID":"30074ef6-2972-4a1d-ad80-fa6f1319747e","Type":"ContainerDied","Data":"e01b5c3685da798fa4c1bac89f8fffde99c09b8758a8cbfbf876dcdcadf488b4"}
Sep 17 18:29:56.326238 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.326108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" event={"ID":"30074ef6-2972-4a1d-ad80-fa6f1319747e","Type":"ContainerStarted","Data":"4ce6bc60e5aace554b0c2fe58de2f515d04b1ee811ea15816991dd1d4e167288"}
Sep 17 18:29:56.327263 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.327238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z6jng" event={"ID":"97b2971f-a12f-494b-83aa-6c14a77418fd","Type":"ContainerStarted","Data":"3ce6c36665a50d75e7ca73b1d44d89d9f645956b6b68c2a3d32a92b847145d38"}
Sep 17 18:29:56.328707 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.328669    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c" event={"ID":"c5e409ea-38ae-4695-82a3-470500adce4d","Type":"ContainerStarted","Data":"51dfaacf4a70b47d8a5aacd1e780673d97bbb0e08e7649a3912615298e028ef9"}
Sep 17 18:29:56.329990 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.329967    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8gvdb" event={"ID":"7a825fbc-a453-4665-a358-cd2d1b343a3b","Type":"ContainerStarted","Data":"f5084fc5804dbb724c3482cf1d20da0214303920b6684f6beee2123a46675449"}
Sep 17 18:29:56.331394 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.331378    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l6754" event={"ID":"ab0ee06f-fab8-477c-a4a2-871e2d44308b","Type":"ContainerStarted","Data":"53392e13c7edb5d66886656deac8fbebfecb6736c35a7472e50b14c4ba43e3b0"}
Sep 17 18:29:56.332748 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.332725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xdtn6" event={"ID":"22c6c31c-3640-4498-bf6f-4023379af97a","Type":"ContainerStarted","Data":"6e67483f1731d7b01f0d759be533b766178c3ea209254914ded1460697bfc1b7"}
Sep 17 18:29:56.334187 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.334167    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdb005e3-9caa-4fd7-ae41-bc8936506eb2" containerID="0e3cf24a12863afa65dc660bce6ef07fe7b14e5275294798afcef3ce4e5d68ae" exitCode=0
Sep 17 18:29:56.334255 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.334233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sbr" event={"ID":"cdb005e3-9caa-4fd7-ae41-bc8936506eb2","Type":"ContainerDied","Data":"0e3cf24a12863afa65dc660bce6ef07fe7b14e5275294798afcef3ce4e5d68ae"}
Sep 17 18:29:56.341272 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.341233    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z6jng" podStartSLOduration=4.464808963 podStartE2EDuration="21.341220954s" totalImagesPullingTime="16.876411991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:56.340662903 +0000 UTC m=+23.620319913" watchObservedRunningTime="2026-09-17 18:29:56.341220954 +0000 UTC m=+23.620877982"
Sep 17 18:29:56.356801 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.356764    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l6754" podStartSLOduration=4.337358988 podStartE2EDuration="21.356742601s" totalImagesPullingTime="17.019383613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:56.356695914 +0000 UTC m=+23.636352944" watchObservedRunningTime="2026-09-17 18:29:56.356742601 +0000 UTC m=+23.636399611"
Sep 17 18:29:56.373162 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.373127    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xdtn6" podStartSLOduration=4.503978107 podStartE2EDuration="21.373115893s" totalImagesPullingTime="16.869137786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:56.37284137 +0000 UTC m=+23.652498412" watchObservedRunningTime="2026-09-17 18:29:56.373115893 +0000 UTC m=+23.652772964"
Sep 17 18:29:56.395609 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.394158    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8gvdb" podStartSLOduration=4.317082606 podStartE2EDuration="21.394142672s" totalImagesPullingTime="17.077060066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:56.393235398 +0000 UTC m=+23.672892429" watchObservedRunningTime="2026-09-17 18:29:56.394142672 +0000 UTC m=+23.673799701"
Sep 17 18:29:56.515878 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.515838    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:56.575805 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.575780    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:56.653916 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:56.653881    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:29:57.122625 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.122521    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:29:56.653913502Z","UUID":"2a628d38-6f63-44ea-94be-eaae9a8a32b8","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:29:57.124199 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.124179    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:29:57.124199 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.124214    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:29:57.213399 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.213370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:57.213597 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.213374    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:57.213597 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:57.213539    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:57.213597 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.213580    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:57.213770 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:57.213680    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:57.213770 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:57.213754    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:57.338790 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.338756    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c" event={"ID":"c5e409ea-38ae-4695-82a3-470500adce4d","Type":"ContainerStarted","Data":"386b17b192c9d9990483e1479cc09117f8182d9b47ad01365ac1f7b7ac8b4c66"}
Sep 17 18:29:57.340576 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.340547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zxrrd" event={"ID":"022b9b32-4714-42a9-b5f7-542385cc4352","Type":"ContainerStarted","Data":"ac4b6e0bcca8deb64ed7f4287d469be382ee818230a48d81588341a1cc84e25f"}
Sep 17 18:29:57.356647 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.356605    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zxrrd" podStartSLOduration=5.332398721 podStartE2EDuration="22.35658742s" totalImagesPullingTime="17.024188699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:57.356241004 +0000 UTC m=+24.635898036" watchObservedRunningTime="2026-09-17 18:29:57.35658742 +0000 UTC m=+24.636244451"
Sep 17 18:29:57.477246 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.477174    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:57.578598 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:57.578565    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:58.100653 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:58.100616    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:58.170005 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:58.169967    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:58.347358 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:58.347331    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:29:58.347826 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:58.347772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" event={"ID":"30074ef6-2972-4a1d-ad80-fa6f1319747e","Type":"ContainerStarted","Data":"79005a0b8d2d8f1084ca07af491a165b24829c64497ad45a58af667357b53ff3"}
Sep 17 18:29:58.350111 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:58.350086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c" event={"ID":"c5e409ea-38ae-4695-82a3-470500adce4d","Type":"ContainerStarted","Data":"a0f1a131efd4a4080437602fd576a560c88a4fca852a18db7016d573e4c08843"}
Sep 17 18:29:58.374640 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:58.374599    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bw4c" podStartSLOduration=4.467328343 podStartE2EDuration="23.374583854s" totalImagesPullingTime="18.907255511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:58.373909814 +0000 UTC m=+25.653566841" watchObservedRunningTime="2026-09-17 18:29:58.374583854 +0000 UTC m=+25.654240884"
Sep 17 18:29:58.676369 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:58.676285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:59.063850 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:59.063819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:59.163988 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:59.163937    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:59.212959 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:59.212933    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:29:59.212959 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:59.212949    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:29:59.213196 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:59.212933    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:29:59.213196 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:59.213061    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:29:59.213196 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:59.213128    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:29:59.213336 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:29:59.213216    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:29:59.800314 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:59.800286    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:29:59.800966 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:29:59.800944    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:30:00.358274 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:00.358121    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:30:00.358687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:00.358658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" event={"ID":"30074ef6-2972-4a1d-ad80-fa6f1319747e","Type":"ContainerStarted","Data":"512538d7d58236ce2830e429ea9c930c9aa5edfbe6f11339255b420a03ed5847"}
Sep 17 18:30:00.358791 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:00.358779    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:30:00.359370 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:00.359287    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4d7m6"
Sep 17 18:30:00.428515 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:00.428465    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:01.213142 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:01.213113    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:01.213888 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:01.213113    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:30:01.213888 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:01.213221    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:30:01.213888 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:01.213113    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:30:01.213888 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:01.213267    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:30:01.213888 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:01.213361    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:30:01.362569 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:01.362535    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdb005e3-9caa-4fd7-ae41-bc8936506eb2" containerID="5fbb152a4ff03360c3d8872288d1ae8681bf29c01cf4d32da4b59d240dd7b88c" exitCode=0
Sep 17 18:30:01.362730 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:01.362617    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sbr" event={"ID":"cdb005e3-9caa-4fd7-ae41-bc8936506eb2","Type":"ContainerDied","Data":"5fbb152a4ff03360c3d8872288d1ae8681bf29c01cf4d32da4b59d240dd7b88c"}
Sep 17 18:30:01.363080 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:01.363061    2561 scope.go:117] "RemoveContainer" containerID="e01b5c3685da798fa4c1bac89f8fffde99c09b8758a8cbfbf876dcdcadf488b4"
Sep 17 18:30:02.203166 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.202997    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.228470 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.228434    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p5hr5"]
Sep 17 18:30:02.228900 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.228680    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:30:02.228900 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:02.228798    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:30:02.229192 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.229176    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-szc74"]
Sep 17 18:30:02.229274 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.229262    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:02.229350 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:02.229335    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:30:02.231057 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.231034    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w4s89"]
Sep 17 18:30:02.231159 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.231133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:30:02.231242 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:02.231223    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:30:02.371918 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.371891    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:30:02.372301 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.372269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" event={"ID":"30074ef6-2972-4a1d-ad80-fa6f1319747e","Type":"ContainerStarted","Data":"97149174d57acebea9fcdf195a18f2fae0b75879f3f29855cfeb38c6bd54ccc5"}
Sep 17 18:30:02.372802 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.372628    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:30:02.372802 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.372702    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:30:02.372802 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.372752    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:30:02.387611 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.387590    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:30:02.391867 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.391851    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:30:02.402490 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:02.402436    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv" podStartSLOduration=10.37994623 podStartE2EDuration="27.402422645s" totalImagesPullingTime="17.022476415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:02.401005044 +0000 UTC m=+29.680662069" watchObservedRunningTime="2026-09-17 18:30:02.402422645 +0000 UTC m=+29.682079715"
Sep 17 18:30:03.377091 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:03.377053    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdb005e3-9caa-4fd7-ae41-bc8936506eb2" containerID="39e906a228ecaf1dd17dd20cb57396fde7014c25a011d9a8939c0d175d8655dd" exitCode=0
Sep 17 18:30:03.377546 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:03.377143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sbr" event={"ID":"cdb005e3-9caa-4fd7-ae41-bc8936506eb2","Type":"ContainerDied","Data":"39e906a228ecaf1dd17dd20cb57396fde7014c25a011d9a8939c0d175d8655dd"}
Sep 17 18:30:04.213443 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:04.213414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:30:04.213592 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:04.213414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:30:04.213592 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:04.213545    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:30:04.213720 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:04.213414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:04.213720 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:04.213623    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:30:04.213720 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:04.213673    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:30:05.384724 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:05.384691    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdb005e3-9caa-4fd7-ae41-bc8936506eb2" containerID="6a877e82989841acde4b6d172626b8b6c81d2b3a52f2d5f63fcbcf5aeab05e9d" exitCode=0
Sep 17 18:30:05.385061 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:05.384748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sbr" event={"ID":"cdb005e3-9caa-4fd7-ae41-bc8936506eb2","Type":"ContainerDied","Data":"6a877e82989841acde4b6d172626b8b6c81d2b3a52f2d5f63fcbcf5aeab05e9d"}
Sep 17 18:30:05.410440 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:05.410409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:05.466116 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:05.466088    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.213250 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:06.213214    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:30:06.213415 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:06.213231    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:30:06.213415 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:06.213328    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:06.213415 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:06.213328    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:30:06.213415 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:06.213402    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:30:06.213624 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:06.213502    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:30:06.351602 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:06.351574    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.442056 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:06.442021    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.647174 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:06.647140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.723954 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:06.723921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.824909 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:06.824879    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.976708 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:06.976675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:07.078178 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:07.078008    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:07.178329 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:07.178299    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:07.384890 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:07.384857    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:07.451553 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:07.451461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.212522 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:08.212494    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:08.212686 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:08.212540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:30:08.212686 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:08.212541    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:30:08.212686 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.212616    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-szc74" podUID="be355727-2ed7-493b-9067-832bffebf603"
Sep 17 18:30:08.212837 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.212728    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4s89" podUID="e3704c11-e37f-415e-b1dd-1b0facfeaa42"
Sep 17 18:30:08.212889 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.212830    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5hr5" podUID="1213f80e-4a5f-4930-ae2e-2734dad1d892"
Sep 17 18:30:08.388493 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:08.388438    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z6jng_97b2971f-a12f-494b-83aa-6c14a77418fd/dns-node-resolver/0.log"
Sep 17 18:30:08.593289 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:08.593257    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.727288 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:08.727259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.787340 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:08.787314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:30:08.787521 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:08.787349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:30:08.787521 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.787462    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:30:08.787680 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.787543    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret podName:e3704c11-e37f-415e-b1dd-1b0facfeaa42 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:40.787524466 +0000 UTC m=+68.067181488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret") pod "global-pull-secret-syncer-w4s89" (UID: "e3704c11-e37f-415e-b1dd-1b0facfeaa42") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:30:08.787680 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.787462    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:30:08.787680 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.787596    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs podName:1213f80e-4a5f-4930-ae2e-2734dad1d892 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:40.787584337 +0000 UTC m=+68.067241344 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs") pod "network-metrics-daemon-p5hr5" (UID: "1213f80e-4a5f-4930-ae2e-2734dad1d892") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:30:08.801241 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:08.801187    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.887729 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:08.887688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676p9\" (UniqueName: \"kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9\") pod \"network-check-target-szc74\" (UID: \"be355727-2ed7-493b-9067-832bffebf603\") " pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:08.887882 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.887864    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:30:08.887947 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.887887    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:30:08.887947 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.887896    2561 projected.go:196] Error preparing data for projected volume kube-api-access-676p9 for pod openshift-network-diagnostics/network-check-target-szc74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:30:08.888046 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:08.887953    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9 podName:be355727-2ed7-493b-9067-832bffebf603 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:40.887939699 +0000 UTC m=+68.167596727 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-676p9" (UniqueName: "kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9") pod "network-check-target-szc74" (UID: "be355727-2ed7-493b-9067-832bffebf603") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:30:09.037460 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.037433    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeReady"
Sep 17 18:30:09.037650 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.037591    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:30:09.105977 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.105948    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c766f"]
Sep 17 18:30:09.132992 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.132925    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6vfht"]
Sep 17 18:30:09.133124 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.133040    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.142274 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.142256    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:30:09.142388 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.142274    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f4hc6\""
Sep 17 18:30:09.142388 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.142305    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:30:09.151161 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.151139    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c766f"]
Sep 17 18:30:09.151265 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.151179    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6vfht"]
Sep 17 18:30:09.151265 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.151252    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6vfht"
Sep 17 18:30:09.155003 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.154844    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:30:09.155124 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.155104    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:30:09.155600 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.155581    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:30:09.155600 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.155595    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qrnqw\""
Sep 17 18:30:09.291099 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.291068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f89ea57-8bac-4b31-9331-f76a08fed0f4-config-volume\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.291099 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.291103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4mh5\" (UniqueName: \"kubernetes.io/projected/3aae0230-f1ff-48ca-b9b2-edf0dfe312ed-kube-api-access-r4mh5\") pod \"ingress-canary-6vfht\" (UID: \"3aae0230-f1ff-48ca-b9b2-edf0dfe312ed\") " pod="openshift-ingress-canary/ingress-canary-6vfht"
Sep 17 18:30:09.291339 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.291121    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3aae0230-f1ff-48ca-b9b2-edf0dfe312ed-cert\") pod \"ingress-canary-6vfht\" (UID: \"3aae0230-f1ff-48ca-b9b2-edf0dfe312ed\") " pod="openshift-ingress-canary/ingress-canary-6vfht"
Sep 17 18:30:09.291339 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.291141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f89ea57-8bac-4b31-9331-f76a08fed0f4-tmp-dir\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.291339 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.291204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.291339 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.291236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk49k\" (UniqueName: \"kubernetes.io/projected/5f89ea57-8bac-4b31-9331-f76a08fed0f4-kube-api-access-bk49k\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.386879 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.386815    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xdtn6_22c6c31c-3640-4498-bf6f-4023379af97a/node-ca/0.log"
Sep 17 18:30:09.392060 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.392032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk49k\" (UniqueName: \"kubernetes.io/projected/5f89ea57-8bac-4b31-9331-f76a08fed0f4-kube-api-access-bk49k\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.392199 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.392125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f89ea57-8bac-4b31-9331-f76a08fed0f4-config-volume\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.392199 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.392164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mh5\" (UniqueName: \"kubernetes.io/projected/3aae0230-f1ff-48ca-b9b2-edf0dfe312ed-kube-api-access-r4mh5\") pod \"ingress-canary-6vfht\" (UID: \"3aae0230-f1ff-48ca-b9b2-edf0dfe312ed\") " pod="openshift-ingress-canary/ingress-canary-6vfht"
Sep 17 18:30:09.392199 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.392185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3aae0230-f1ff-48ca-b9b2-edf0dfe312ed-cert\") pod \"ingress-canary-6vfht\" (UID: \"3aae0230-f1ff-48ca-b9b2-edf0dfe312ed\") " pod="openshift-ingress-canary/ingress-canary-6vfht"
Sep 17 18:30:09.392314 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.392207    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f89ea57-8bac-4b31-9331-f76a08fed0f4-tmp-dir\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.392314 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.392224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.392453 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:09.392425    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:09.392564 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.392537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f89ea57-8bac-4b31-9331-f76a08fed0f4-tmp-dir\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.392685 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:09.392543    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls podName:5f89ea57-8bac-4b31-9331-f76a08fed0f4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.892520276 +0000 UTC m=+37.172177299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls") pod "dns-default-c766f" (UID: "5f89ea57-8bac-4b31-9331-f76a08fed0f4") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:09.392759 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.392698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f89ea57-8bac-4b31-9331-f76a08fed0f4-config-volume\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.396476 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.396449    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3aae0230-f1ff-48ca-b9b2-edf0dfe312ed-cert\") pod \"ingress-canary-6vfht\" (UID: \"3aae0230-f1ff-48ca-b9b2-edf0dfe312ed\") " pod="openshift-ingress-canary/ingress-canary-6vfht"
Sep 17 18:30:09.403796 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.403777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk49k\" (UniqueName: \"kubernetes.io/projected/5f89ea57-8bac-4b31-9331-f76a08fed0f4-kube-api-access-bk49k\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.411344 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.411323    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4mh5\" (UniqueName: \"kubernetes.io/projected/3aae0230-f1ff-48ca-b9b2-edf0dfe312ed-kube-api-access-r4mh5\") pod \"ingress-canary-6vfht\" (UID: \"3aae0230-f1ff-48ca-b9b2-edf0dfe312ed\") " pod="openshift-ingress-canary/ingress-canary-6vfht"
Sep 17 18:30:09.461901 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.461878    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6vfht"
Sep 17 18:30:09.573308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.573274    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:09.782921 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.782900    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6vfht"]
Sep 17 18:30:09.786259 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:30:09.786236    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aae0230_f1ff_48ca_b9b2_edf0dfe312ed.slice/crio-395158c15082b451743e6255d02757c4052e161e52ce49fb8628b7a3f9c34a4b WatchSource:0}: Error finding container 395158c15082b451743e6255d02757c4052e161e52ce49fb8628b7a3f9c34a4b: Status 404 returned error can't find the container with id 395158c15082b451743e6255d02757c4052e161e52ce49fb8628b7a3f9c34a4b
Sep 17 18:30:09.895877 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:09.895712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:09.895877 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:09.895866    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:09.896038 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:09.895976    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls podName:5f89ea57-8bac-4b31-9331-f76a08fed0f4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.895957887 +0000 UTC m=+38.175614894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls") pod "dns-default-c766f" (UID: "5f89ea57-8bac-4b31-9331-f76a08fed0f4") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:10.213188 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.213163    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:30:10.213359 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.213165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:30:10.213405 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.213165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:10.218284 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.218103    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:10.218284 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.218109    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:10.219400 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.219238    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bhngp\""
Sep 17 18:30:10.219400 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.219273    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:30:10.219400 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.219271    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:30:10.219833 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.219529    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmb9h\""
Sep 17 18:30:10.398095 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.398054    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6vfht" event={"ID":"3aae0230-f1ff-48ca-b9b2-edf0dfe312ed","Type":"ContainerStarted","Data":"395158c15082b451743e6255d02757c4052e161e52ce49fb8628b7a3f9c34a4b"}
Sep 17 18:30:10.401464 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.401437    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdb005e3-9caa-4fd7-ae41-bc8936506eb2" containerID="350bca58df8b74c2482a0467e85d5854fb261e9484cd98809fd87bfa0f167291" exitCode=0
Sep 17 18:30:10.401615 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.401505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sbr" event={"ID":"cdb005e3-9caa-4fd7-ae41-bc8936506eb2","Type":"ContainerDied","Data":"350bca58df8b74c2482a0467e85d5854fb261e9484cd98809fd87bfa0f167291"}
Sep 17 18:30:10.902082 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:10.902042    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:10.902604 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:10.902236    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:10.902604 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:10.902323    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls podName:5f89ea57-8bac-4b31-9331-f76a08fed0f4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:12.902300113 +0000 UTC m=+40.181957127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls") pod "dns-default-c766f" (UID: "5f89ea57-8bac-4b31-9331-f76a08fed0f4") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:11.407699 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:11.407660    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdb005e3-9caa-4fd7-ae41-bc8936506eb2" containerID="b6f44d5b6a8974fe74f3195ab7824d9ef81aba8c3596d35e27d86e46104dfb03" exitCode=0
Sep 17 18:30:11.407699 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:11.407709    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sbr" event={"ID":"cdb005e3-9caa-4fd7-ae41-bc8936506eb2","Type":"ContainerDied","Data":"b6f44d5b6a8974fe74f3195ab7824d9ef81aba8c3596d35e27d86e46104dfb03"}
Sep 17 18:30:12.039648 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.039618    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.248441 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.248414    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.362716 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.362684    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.413654 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.413614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6sbr" event={"ID":"cdb005e3-9caa-4fd7-ae41-bc8936506eb2","Type":"ContainerStarted","Data":"cadbc6258a45c286dd4eee4786275497803377454509f505015462f8426cf590"}
Sep 17 18:30:12.414908 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.414886    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6vfht" event={"ID":"3aae0230-f1ff-48ca-b9b2-edf0dfe312ed","Type":"ContainerStarted","Data":"5118cb2c19a769991840b9f322d12e632ab3d9c9f4ec624eb93bf577c9654164"}
Sep 17 18:30:12.442045 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.442002    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c6sbr" podStartSLOduration=9.294684799 podStartE2EDuration="37.441987737s" totalImagesPullingTime="28.147302938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:12.440065813 +0000 UTC m=+39.719722840" watchObservedRunningTime="2026-09-17 18:30:12.441987737 +0000 UTC m=+39.721644765"
Sep 17 18:30:12.449102 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.449076    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.461488 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.461446    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6vfht" podStartSLOduration=1.272631476 podStartE2EDuration="3.461432771s" totalImagesPullingTime="2.188801295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:12.460237088 +0000 UTC m=+39.739894116" watchObservedRunningTime="2026-09-17 18:30:12.461432771 +0000 UTC m=+39.741089799"
Sep 17 18:30:12.550894 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.550868    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.727794 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.727721    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.828082 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.828053    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.915536 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:12.915505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:12.915665 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:12.915610    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:12.915665 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:30:12.915657    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls podName:5f89ea57-8bac-4b31-9331-f76a08fed0f4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:16.915645729 +0000 UTC m=+44.195302735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls") pod "dns-default-c766f" (UID: "5f89ea57-8bac-4b31-9331-f76a08fed0f4") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:13.018313 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:13.018244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.537427 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:13.537402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.740414 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:13.740386    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.147950 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:14.147919    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.741561 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:14.741526    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:15.165407 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:15.165383    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:15.558918 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:15.558892    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:15.732093 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:15.732068    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:15.834052 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:15.833975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.032891 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:16.032856    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.336589 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:16.336557    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.944838 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:16.944809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:16.947306 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:16.947279    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.948365 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:16.948343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f89ea57-8bac-4b31-9331-f76a08fed0f4-metrics-tls\") pod \"dns-default-c766f\" (UID: \"5f89ea57-8bac-4b31-9331-f76a08fed0f4\") " pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:17.009190 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:17.009166    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:17.140328 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:17.140301    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:17.243806 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:17.243745    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:17.339346 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:17.339314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:17.520914 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:17.520850    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c766f"]
Sep 17 18:30:17.523797 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:30:17.523774    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f89ea57_8bac_4b31_9331_f76a08fed0f4.slice/crio-5d062fed7ed8d9dbbcc6dac9a68997972205c557091c87979fd1f7d54530a3af WatchSource:0}: Error finding container 5d062fed7ed8d9dbbcc6dac9a68997972205c557091c87979fd1f7d54530a3af: Status 404 returned error can't find the container with id 5d062fed7ed8d9dbbcc6dac9a68997972205c557091c87979fd1f7d54530a3af
Sep 17 18:30:17.733870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:17.733841    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.137677 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:18.137643    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.399122 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:18.399049    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.431395 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:18.431366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c766f" event={"ID":"5f89ea57-8bac-4b31-9331-f76a08fed0f4","Type":"ContainerStarted","Data":"5d062fed7ed8d9dbbcc6dac9a68997972205c557091c87979fd1f7d54530a3af"}
Sep 17 18:30:18.739509 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:18.739416    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.133536 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:19.133502    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.337331 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:19.337299    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.436146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:19.436107    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c766f" event={"ID":"5f89ea57-8bac-4b31-9331-f76a08fed0f4","Type":"ContainerStarted","Data":"3d2a6a5bc4e78e6d60ce2ac60ef88a4d5d0d95bff0343ccd9ba94f578e9ec6f3"}
Sep 17 18:30:19.436314 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:19.436164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c766f" event={"ID":"5f89ea57-8bac-4b31-9331-f76a08fed0f4","Type":"ContainerStarted","Data":"ce92f50e3e838d222ae839a12210737f48d4621530bd31df13ab73a42422c983"}
Sep 17 18:30:19.436314 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:19.436237    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:19.454113 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:19.454072    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c766f" podStartSLOduration=9.104725621 podStartE2EDuration="10.454057113s" totalImagesPullingTime="1.349331492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:19.453465681 +0000 UTC m=+46.733122710" watchObservedRunningTime="2026-09-17 18:30:19.454057113 +0000 UTC m=+46.733714140"
Sep 17 18:30:19.735900 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:19.735816    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.120294 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:20.120267    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.544604 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:20.544530    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.748920 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:20.748893    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.134242 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:21.134206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.341864 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:21.341837    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.619537 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:21.619513    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.914603 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:21.914530    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:22.149368 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:22.149338    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:22.556453 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:22.556421    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:22.761669 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:22.761635    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:23.653076 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:23.653045    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:23.754844 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:23.754819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:23.832339 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:23.832316    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:23.950521 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:23.950443    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.141078 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:24.141047    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.219735 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:24.219670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.524624 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:24.524547    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.727031 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:24.727002    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.830852 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:24.830827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.931310 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:24.931284    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.999833 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:24.999807    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:25.345957 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:25.345925    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:25.540559 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:25.540533    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:26.159621 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:26.159590    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:26.218868 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:26.218843    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:29.440019 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:29.439993    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c766f"
Sep 17 18:30:30.935318 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.935265    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"]
Sep 17 18:30:30.964085 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.964048    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"]
Sep 17 18:30:30.964247 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.964176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"
Sep 17 18:30:30.966728 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.966709    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 18:30:30.967699 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.967672    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-r6rpx\""
Sep 17 18:30:30.967699 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.967696    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 18:30:30.967870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.967696    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 18:30:30.967870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.967771    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 18:30:30.977833 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.977813    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"]
Sep 17 18:30:30.977954 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.977925    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:30.981191 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.981173    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 18:30:30.995912 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.995893    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"]
Sep 17 18:30:30.995996 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.995919    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"]
Sep 17 18:30:30.996033 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:30.996005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.004580 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.004553    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 18:30:31.004679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.004632    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 18:30:31.004679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.004637    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 18:30:31.004779 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.004684    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 18:30:31.005391 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.005372    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"]
Sep 17 18:30:31.048619 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.048595    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cp5ds"]
Sep 17 18:30:31.072113 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.072091    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.086581 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.086554    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:30:31.086841 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.086809    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:30:31.086841 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.086824    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:30:31.087090 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.087044    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:30:31.087259 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.087246    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kvfvs\""
Sep 17 18:30:31.098964 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.098945    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cp5ds"]
Sep 17 18:30:31.124799 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.124766    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:31.141967 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.141942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.142064 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.141973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ba9a930e-e08b-4f35-9927-0330c75e63e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-bd6d9c574-8tzm4\" (UID: \"ba9a930e-e08b-4f35-9927-0330c75e63e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:31.142064 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.142002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84fgd\" (UniqueName: \"kubernetes.io/projected/ba9a930e-e08b-4f35-9927-0330c75e63e7-kube-api-access-84fgd\") pod \"klusterlet-addon-workmgr-bd6d9c574-8tzm4\" (UID: \"ba9a930e-e08b-4f35-9927-0330c75e63e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:31.142175 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.142104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf7vh\" (UniqueName: \"kubernetes.io/projected/6a696272-2120-403c-8052-cf7352a4a8af-kube-api-access-sf7vh\") pod \"managed-serviceaccount-addon-agent-54f44f86c6-j9v47\" (UID: \"6a696272-2120-403c-8052-cf7352a4a8af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"
Sep 17 18:30:31.142175 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.142129    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba9a930e-e08b-4f35-9927-0330c75e63e7-tmp\") pod \"klusterlet-addon-workmgr-bd6d9c574-8tzm4\" (UID: \"ba9a930e-e08b-4f35-9927-0330c75e63e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:31.142175 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.142154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-hub\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.142302 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.142178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9c8\" (UniqueName: \"kubernetes.io/projected/0c3affc2-8d48-4832-8140-12fd0918abba-kube-api-access-ph9c8\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.142302 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.142210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-ca\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.142302 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.142233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a696272-2120-403c-8052-cf7352a4a8af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54f44f86c6-j9v47\" (UID: \"6a696272-2120-403c-8052-cf7352a4a8af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"
Sep 17 18:30:31.142302 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.142287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3affc2-8d48-4832-8140-12fd0918abba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.142425 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.142315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.242786 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242727    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3affc2-8d48-4832-8140-12fd0918abba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.242786 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.242786 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvm89\" (UniqueName: \"kubernetes.io/projected/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-kube-api-access-lvm89\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.242990 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.242990 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ba9a930e-e08b-4f35-9927-0330c75e63e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-bd6d9c574-8tzm4\" (UID: \"ba9a930e-e08b-4f35-9927-0330c75e63e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:31.242990 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.242990 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84fgd\" (UniqueName: \"kubernetes.io/projected/ba9a930e-e08b-4f35-9927-0330c75e63e7-kube-api-access-84fgd\") pod \"klusterlet-addon-workmgr-bd6d9c574-8tzm4\" (UID: \"ba9a930e-e08b-4f35-9927-0330c75e63e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:31.242990 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-data-volume\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.242990 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.242990 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf7vh\" (UniqueName: \"kubernetes.io/projected/6a696272-2120-403c-8052-cf7352a4a8af-kube-api-access-sf7vh\") pod \"managed-serviceaccount-addon-agent-54f44f86c6-j9v47\" (UID: \"6a696272-2120-403c-8052-cf7352a4a8af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"
Sep 17 18:30:31.242990 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.242977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba9a930e-e08b-4f35-9927-0330c75e63e7-tmp\") pod \"klusterlet-addon-workmgr-bd6d9c574-8tzm4\" (UID: \"ba9a930e-e08b-4f35-9927-0330c75e63e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:31.243367 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.243003    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-hub\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.243367 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.243028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9c8\" (UniqueName: \"kubernetes.io/projected/0c3affc2-8d48-4832-8140-12fd0918abba-kube-api-access-ph9c8\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.243367 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.243061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-ca\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.243367 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.243085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a696272-2120-403c-8052-cf7352a4a8af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54f44f86c6-j9v47\" (UID: \"6a696272-2120-403c-8052-cf7352a4a8af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"
Sep 17 18:30:31.243367 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.243108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-crio-socket\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.243629 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.243423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba9a930e-e08b-4f35-9927-0330c75e63e7-tmp\") pod \"klusterlet-addon-workmgr-bd6d9c574-8tzm4\" (UID: \"ba9a930e-e08b-4f35-9927-0330c75e63e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:31.243629 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.243595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3affc2-8d48-4832-8140-12fd0918abba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.245552 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.245534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.245830 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.245811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-ca\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.245872 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.245825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a696272-2120-403c-8052-cf7352a4a8af-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54f44f86c6-j9v47\" (UID: \"6a696272-2120-403c-8052-cf7352a4a8af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"
Sep 17 18:30:31.245931 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.245913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ba9a930e-e08b-4f35-9927-0330c75e63e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-bd6d9c574-8tzm4\" (UID: \"ba9a930e-e08b-4f35-9927-0330c75e63e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:31.245965 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.245946    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.246000 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.245966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0c3affc2-8d48-4832-8140-12fd0918abba-hub\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.256124 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.256106    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf7vh\" (UniqueName: \"kubernetes.io/projected/6a696272-2120-403c-8052-cf7352a4a8af-kube-api-access-sf7vh\") pod \"managed-serviceaccount-addon-agent-54f44f86c6-j9v47\" (UID: \"6a696272-2120-403c-8052-cf7352a4a8af\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"
Sep 17 18:30:31.266416 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.266391    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9c8\" (UniqueName: \"kubernetes.io/projected/0c3affc2-8d48-4832-8140-12fd0918abba-kube-api-access-ph9c8\") pod \"cluster-proxy-proxy-agent-64689bdd7d-m4dmc\" (UID: \"0c3affc2-8d48-4832-8140-12fd0918abba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.266881 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.266860    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84fgd\" (UniqueName: \"kubernetes.io/projected/ba9a930e-e08b-4f35-9927-0330c75e63e7-kube-api-access-84fgd\") pod \"klusterlet-addon-workmgr-bd6d9c574-8tzm4\" (UID: \"ba9a930e-e08b-4f35-9927-0330c75e63e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:31.282551 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.282534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"
Sep 17 18:30:31.292649 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.292627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:31.303798 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.303779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"
Sep 17 18:30:31.314828 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.314808    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:31.344555 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.344495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-crio-socket\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.344555 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.344552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvm89\" (UniqueName: \"kubernetes.io/projected/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-kube-api-access-lvm89\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.344778 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.344591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.344778 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.344620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-data-volume\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.344778 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.344649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.344778 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.344677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-crio-socket\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.345055 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.345037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-data-volume\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.345303 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.345222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.347558 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.347540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.359731 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.359677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvm89\" (UniqueName: \"kubernetes.io/projected/6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f-kube-api-access-lvm89\") pod \"insights-runtime-extractor-cp5ds\" (UID: \"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f\") " pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.388034 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.387598    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cp5ds"
Sep 17 18:30:31.426838 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.426816    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:31.532703 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.532637    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:31.587907 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.587878    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47"]
Sep 17 18:30:31.592163 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:30:31.592134    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a696272_2120_403c_8052_cf7352a4a8af.slice/crio-e51dd5d437c4202054976c54c1c339380c4f549ac7504805198fa582a914c396 WatchSource:0}: Error finding container e51dd5d437c4202054976c54c1c339380c4f549ac7504805198fa582a914c396: Status 404 returned error can't find the container with id e51dd5d437c4202054976c54c1c339380c4f549ac7504805198fa582a914c396
Sep 17 18:30:31.607317 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.607296    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"]
Sep 17 18:30:31.610122 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:30:31.610096    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9a930e_e08b_4f35_9927_0330c75e63e7.slice/crio-82f7fc9bf2b47b3b18073c774d120b0e6b564387d9234b8a0b052a5ee714ef49 WatchSource:0}: Error finding container 82f7fc9bf2b47b3b18073c774d120b0e6b564387d9234b8a0b052a5ee714ef49: Status 404 returned error can't find the container with id 82f7fc9bf2b47b3b18073c774d120b0e6b564387d9234b8a0b052a5ee714ef49
Sep 17 18:30:31.633959 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.633940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc"]
Sep 17 18:30:31.636399 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:30:31.636376    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c3affc2_8d48_4832_8140_12fd0918abba.slice/crio-633208d8f85138bdd8a0a65a0a2b4fd3030f845f97f40b46445729709979a718 WatchSource:0}: Error finding container 633208d8f85138bdd8a0a65a0a2b4fd3030f845f97f40b46445729709979a718: Status 404 returned error can't find the container with id 633208d8f85138bdd8a0a65a0a2b4fd3030f845f97f40b46445729709979a718
Sep 17 18:30:31.649255 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.649233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:31.695288 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:31.695259    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cp5ds"]
Sep 17 18:30:31.698727 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:30:31.698706    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbb4d50_7d43_4d97_9ee4_e3202e1bc73f.slice/crio-43b78bc202102e1d890a5ef3000c2b60f501cbb8810c2566b141f3ff83c2a43d WatchSource:0}: Error finding container 43b78bc202102e1d890a5ef3000c2b60f501cbb8810c2566b141f3ff83c2a43d: Status 404 returned error can't find the container with id 43b78bc202102e1d890a5ef3000c2b60f501cbb8810c2566b141f3ff83c2a43d
Sep 17 18:30:32.472915 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:32.472820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc" event={"ID":"0c3affc2-8d48-4832-8140-12fd0918abba","Type":"ContainerStarted","Data":"633208d8f85138bdd8a0a65a0a2b4fd3030f845f97f40b46445729709979a718"}
Sep 17 18:30:32.474698 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:32.474574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4" event={"ID":"ba9a930e-e08b-4f35-9927-0330c75e63e7","Type":"ContainerStarted","Data":"82f7fc9bf2b47b3b18073c774d120b0e6b564387d9234b8a0b052a5ee714ef49"}
Sep 17 18:30:32.477015 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:32.476951    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47" event={"ID":"6a696272-2120-403c-8052-cf7352a4a8af","Type":"ContainerStarted","Data":"e51dd5d437c4202054976c54c1c339380c4f549ac7504805198fa582a914c396"}
Sep 17 18:30:32.479107 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:32.479008    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cp5ds" event={"ID":"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f","Type":"ContainerStarted","Data":"24b23a8e45005f0a5af2a5d8fe57ecdcc7434b7b23d6a305dfbdd001d9a39658"}
Sep 17 18:30:32.479107 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:32.479055    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cp5ds" event={"ID":"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f","Type":"ContainerStarted","Data":"43b78bc202102e1d890a5ef3000c2b60f501cbb8810c2566b141f3ff83c2a43d"}
Sep 17 18:30:32.568734 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:32.568522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:32.656800 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:32.656763    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:32.901009 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:32.900915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:33.165333 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:33.165264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:33.367678 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:33.367465    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:33.485995 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:33.485916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cp5ds" event={"ID":"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f","Type":"ContainerStarted","Data":"9f8fbc77dafceeceafde38baff06af4dc32c3fd73df21db0099418a387f9d379"}
Sep 17 18:30:34.400634 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:34.400602    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhfjv"
Sep 17 18:30:34.588225 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:34.588194    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:34.676545 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:34.676439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.097047 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:36.097015    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.180439 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:36.180400    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.594304 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:36.594269    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.697407 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:36.697381    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.796642 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:36.796613    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.505584 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.505505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47" event={"ID":"6a696272-2120-403c-8052-cf7352a4a8af","Type":"ContainerStarted","Data":"481c5899d365b33e9bfc7f4cf849d83cb436113922be02f262acf49173fc9b44"}
Sep 17 18:30:38.507584 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.507561    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cp5ds" event={"ID":"6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f","Type":"ContainerStarted","Data":"29fcade6092f640dbf5b390451f215a310d4ce8fc320e5bd2a46276424d700d6"}
Sep 17 18:30:38.509587 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.509564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc" event={"ID":"0c3affc2-8d48-4832-8140-12fd0918abba","Type":"ContainerStarted","Data":"65463ca20d40c28eae7cd5f34f3d2ad39fef01ac0e50805650a0fc59b274ceb4"}
Sep 17 18:30:38.512637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.509831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc" event={"ID":"0c3affc2-8d48-4832-8140-12fd0918abba","Type":"ContainerStarted","Data":"96deadec9d7d81b8f5bac4d8ec83b4c5112f7695085a235e0167a37d7a17c6e5"}
Sep 17 18:30:38.512637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.509874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc" event={"ID":"0c3affc2-8d48-4832-8140-12fd0918abba","Type":"ContainerStarted","Data":"ef9be845ddcf838358fcb739b3814906c1ef47545549373279590692f474d236"}
Sep 17 18:30:38.514215 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.514192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4" event={"ID":"ba9a930e-e08b-4f35-9927-0330c75e63e7","Type":"ContainerStarted","Data":"0fd94e8f5e252f0ab2b9573ad432cf244b0f72a8f289112ef09cc25312b72dd1"}
Sep 17 18:30:38.514348 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.514334    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:38.516011 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.515993    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4"
Sep 17 18:30:38.524995 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.524958    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f44f86c6-j9v47" podStartSLOduration=1.967645778 podStartE2EDuration="8.524946309s" totalImagesPullingTime="6.557300531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:38.524219386 +0000 UTC m=+65.803876413" watchObservedRunningTime="2026-09-17 18:30:38.524946309 +0000 UTC m=+65.804603331"
Sep 17 18:30:38.543749 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.543710    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bd6d9c574-8tzm4" podStartSLOduration=1.979880642 podStartE2EDuration="8.543700084s" totalImagesPullingTime="6.563819442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:38.542891283 +0000 UTC m=+65.822548311" watchObservedRunningTime="2026-09-17 18:30:38.543700084 +0000 UTC m=+65.823357111"
Sep 17 18:30:38.594093 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.594042    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64689bdd7d-m4dmc" podStartSLOduration=2.05632698 podStartE2EDuration="8.594024683s" totalImagesPullingTime="6.537697703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:38.592959647 +0000 UTC m=+65.872616675" watchObservedRunningTime="2026-09-17 18:30:38.594024683 +0000 UTC m=+65.873681714"
Sep 17 18:30:38.626672 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:38.626623    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cp5ds" podStartSLOduration=1.400479182 podStartE2EDuration="7.626604714s" totalImagesPullingTime="6.226125532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:38.626080433 +0000 UTC m=+65.905737462" watchObservedRunningTime="2026-09-17 18:30:38.626604714 +0000 UTC m=+65.906261743"
Sep 17 18:30:39.144489 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:39.144457    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:39.245248 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:39.245220    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.150927 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.150894    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.229934 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.229873    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.572986 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.572955    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.667735 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.667708    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.822862 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.822830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:30:40.822862 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.822863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:30:40.825333 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.825283    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:40.826135 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.826119    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:40.835818 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.835800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3704c11-e37f-415e-b1dd-1b0facfeaa42-original-pull-secret\") pod \"global-pull-secret-syncer-w4s89\" (UID: \"e3704c11-e37f-415e-b1dd-1b0facfeaa42\") " pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:30:40.835942 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.835927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1213f80e-4a5f-4930-ae2e-2734dad1d892-metrics-certs\") pod \"network-metrics-daemon-p5hr5\" (UID: \"1213f80e-4a5f-4930-ae2e-2734dad1d892\") " pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:30:40.923166 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.923140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676p9\" (UniqueName: \"kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9\") pod \"network-check-target-szc74\" (UID: \"be355727-2ed7-493b-9067-832bffebf603\") " pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:40.925776 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.925756    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:30:40.935752 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.935726    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:30:40.945943 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:40.945915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-676p9\" (UniqueName: \"kubernetes.io/projected/be355727-2ed7-493b-9067-832bffebf603-kube-api-access-676p9\") pod \"network-check-target-szc74\" (UID: \"be355727-2ed7-493b-9067-832bffebf603\") " pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:41.125820 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.125746    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmb9h\""
Sep 17 18:30:41.127805 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.127791    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4s89"
Sep 17 18:30:41.133128 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.133110    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5hr5"
Sep 17 18:30:41.134140 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.134121    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bhngp\""
Sep 17 18:30:41.142335 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.142314    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:41.403334 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.403183    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w4s89"]
Sep 17 18:30:41.404311 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.404281    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p5hr5"]
Sep 17 18:30:41.406100 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:30:41.406080    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3704c11_e37f_415e_b1dd_1b0facfeaa42.slice/crio-68e63729181e6ef397266abe5e566884037fc7c12a61d19cd95db86042d6355c WatchSource:0}: Error finding container 68e63729181e6ef397266abe5e566884037fc7c12a61d19cd95db86042d6355c: Status 404 returned error can't find the container with id 68e63729181e6ef397266abe5e566884037fc7c12a61d19cd95db86042d6355c
Sep 17 18:30:41.407723 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:30:41.407701    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1213f80e_4a5f_4930_ae2e_2734dad1d892.slice/crio-a213eaaf42a7570b3c73a3e18a06a4d49ac7a2386aeef4394dde1589606fc17b WatchSource:0}: Error finding container a213eaaf42a7570b3c73a3e18a06a4d49ac7a2386aeef4394dde1589606fc17b: Status 404 returned error can't find the container with id a213eaaf42a7570b3c73a3e18a06a4d49ac7a2386aeef4394dde1589606fc17b
Sep 17 18:30:41.428307 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.428284    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-szc74"]
Sep 17 18:30:41.432279 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:30:41.432256    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe355727_2ed7_493b_9067_832bffebf603.slice/crio-24f3b301e85aa6b9292a3d66222cdfd7388b9f214eb94352cab9313ca8d5ed6c WatchSource:0}: Error finding container 24f3b301e85aa6b9292a3d66222cdfd7388b9f214eb94352cab9313ca8d5ed6c: Status 404 returned error can't find the container with id 24f3b301e85aa6b9292a3d66222cdfd7388b9f214eb94352cab9313ca8d5ed6c
Sep 17 18:30:41.514429 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.514402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:41.523761 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.523738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p5hr5" event={"ID":"1213f80e-4a5f-4930-ae2e-2734dad1d892","Type":"ContainerStarted","Data":"a213eaaf42a7570b3c73a3e18a06a4d49ac7a2386aeef4394dde1589606fc17b"}
Sep 17 18:30:41.524958 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.524938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w4s89" event={"ID":"e3704c11-e37f-415e-b1dd-1b0facfeaa42","Type":"ContainerStarted","Data":"68e63729181e6ef397266abe5e566884037fc7c12a61d19cd95db86042d6355c"}
Sep 17 18:30:41.526049 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:41.526030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-szc74" event={"ID":"be355727-2ed7-493b-9067-832bffebf603","Type":"ContainerStarted","Data":"24f3b301e85aa6b9292a3d66222cdfd7388b9f214eb94352cab9313ca8d5ed6c"}
Sep 17 18:30:42.553793 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:42.553760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:42.664202 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:42.664150    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:43.324399 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.324364    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:43.537858 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.537825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p5hr5" event={"ID":"1213f80e-4a5f-4930-ae2e-2734dad1d892","Type":"ContainerStarted","Data":"3d43da8cd0c20ab399f822ab21eeb1f07195488c516ed9c1fd36a318f9c41df1"}
Sep 17 18:30:43.538032 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.537883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p5hr5" event={"ID":"1213f80e-4a5f-4930-ae2e-2734dad1d892","Type":"ContainerStarted","Data":"72de47850e57df7abd78be42e19577d0dfbce339fbc13b066542b6a1fd3c14bc"}
Sep 17 18:30:43.577547 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.577444    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p5hr5" podStartSLOduration=67.489732281 podStartE2EDuration="1m8.577427502s" totalImagesPullingTime="1.087695221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:43.575812726 +0000 UTC m=+70.855469753" watchObservedRunningTime="2026-09-17 18:30:43.577427502 +0000 UTC m=+70.857084530"
Sep 17 18:30:43.813950 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.813918    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:43.844885 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.844809    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5nvz6"]
Sep 17 18:30:43.848814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.848785    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:43.857946 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.857922    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:30:43.858182 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.858165    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:30:43.858262 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.858195    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:30:43.858455 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.858439    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:30:43.859021 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.859006    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:30:43.863720 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.863524    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-csfkx\""
Sep 17 18:30:43.863720 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.863556    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:30:43.948104 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.948061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f609f65-9e6b-42fc-bc03-2809aed32e44-sys\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:43.948255 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.948118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-accelerators-collector-config\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:43.948255 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.948147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-tls\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:43.948255 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.948164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f609f65-9e6b-42fc-bc03-2809aed32e44-metrics-client-ca\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:43.948255 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.948181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f609f65-9e6b-42fc-bc03-2809aed32e44-root\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:43.948255 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.948244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-wtmp\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:43.948418 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.948301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-textfile\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:43.948418 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.948323    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84qx\" (UniqueName: \"kubernetes.io/projected/0f609f65-9e6b-42fc-bc03-2809aed32e44-kube-api-access-l84qx\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:43.948418 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.948353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:43.967065 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:43.967036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:44.049426 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-wtmp\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.049426 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-textfile\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.049679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84qx\" (UniqueName: \"kubernetes.io/projected/0f609f65-9e6b-42fc-bc03-2809aed32e44-kube-api-access-l84qx\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.049679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.049679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f609f65-9e6b-42fc-bc03-2809aed32e44-sys\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.049679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-accelerators-collector-config\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.049679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-tls\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.049679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-wtmp\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.049679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f609f65-9e6b-42fc-bc03-2809aed32e44-metrics-client-ca\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.049679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f609f65-9e6b-42fc-bc03-2809aed32e44-root\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.050053 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f609f65-9e6b-42fc-bc03-2809aed32e44-sys\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.050053 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f609f65-9e6b-42fc-bc03-2809aed32e44-root\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.050053 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.049819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-textfile\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.050355 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.050331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-accelerators-collector-config\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.050355 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.050350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f609f65-9e6b-42fc-bc03-2809aed32e44-metrics-client-ca\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.052429 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.052404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.052532 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.052451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f609f65-9e6b-42fc-bc03-2809aed32e44-node-exporter-tls\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.060232 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.060211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84qx\" (UniqueName: \"kubernetes.io/projected/0f609f65-9e6b-42fc-bc03-2809aed32e44-kube-api-access-l84qx\") pod \"node-exporter-5nvz6\" (UID: \"0f609f65-9e6b-42fc-bc03-2809aed32e44\") " pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.089536 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.089510    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:44.160652 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.160572    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5nvz6"
Sep 17 18:30:44.253824 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.253782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:44.576181 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.576153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:44.722322 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.722286    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:44.854699 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:44.854623    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:45.008910 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:45.008872    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:45.775149 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:30:45.775115    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f609f65_9e6b_42fc_bc03_2809aed32e44.slice/crio-54ce356627f4183ba14a7a81e9a1c3a0b68c2a6d746dd4cb1119a003c23c705e WatchSource:0}: Error finding container 54ce356627f4183ba14a7a81e9a1c3a0b68c2a6d746dd4cb1119a003c23c705e: Status 404 returned error can't find the container with id 54ce356627f4183ba14a7a81e9a1c3a0b68c2a6d746dd4cb1119a003c23c705e
Sep 17 18:30:46.556087 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:46.555944    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-szc74" event={"ID":"be355727-2ed7-493b-9067-832bffebf603","Type":"ContainerStarted","Data":"fb986bb0f1518c48bb34ebfdc0beb3a85aa131aeb51077c6644ae9718a8008ba"}
Sep 17 18:30:46.556087 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:46.556053    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:30:46.557883 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:46.557850    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w4s89" event={"ID":"e3704c11-e37f-415e-b1dd-1b0facfeaa42","Type":"ContainerStarted","Data":"e76641f24457bbbf5040084ab9b4e535a6473c1c9d59f94c64b68a16a3b46842"}
Sep 17 18:30:46.559163 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:46.559139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5nvz6" event={"ID":"0f609f65-9e6b-42fc-bc03-2809aed32e44","Type":"ContainerStarted","Data":"54ce356627f4183ba14a7a81e9a1c3a0b68c2a6d746dd4cb1119a003c23c705e"}
Sep 17 18:30:46.577168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:46.575691    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-szc74" podStartSLOduration=67.229109274 podStartE2EDuration="1m11.575674275s" totalImagesPullingTime="4.346565001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:46.573535672 +0000 UTC m=+73.853192700" watchObservedRunningTime="2026-09-17 18:30:46.575674275 +0000 UTC m=+73.855331307"
Sep 17 18:30:46.591626 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:46.591576    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w4s89" podStartSLOduration=67.21438013 podStartE2EDuration="1m11.591560607s" totalImagesPullingTime="4.377180477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:46.591108057 +0000 UTC m=+73.870765085" watchObservedRunningTime="2026-09-17 18:30:46.591560607 +0000 UTC m=+73.871217635"
Sep 17 18:30:46.626823 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:46.626798    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:46.736036 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:46.735999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:46.850138 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:46.850069    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:47.261547 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:47.261415    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:47.564116 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:47.564057    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f609f65-9e6b-42fc-bc03-2809aed32e44" containerID="7518255df3275f0981bcab1a9b90101389c5021de3a5dfd1d7fef7ce249ca425" exitCode=0
Sep 17 18:30:47.564281 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:47.564144    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5nvz6" event={"ID":"0f609f65-9e6b-42fc-bc03-2809aed32e44","Type":"ContainerDied","Data":"7518255df3275f0981bcab1a9b90101389c5021de3a5dfd1d7fef7ce249ca425"}
Sep 17 18:30:48.265139 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:48.265101    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:48.360455 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:48.360423    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:48.567146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:48.567117    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:48.570019 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:48.569991    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5nvz6" event={"ID":"0f609f65-9e6b-42fc-bc03-2809aed32e44","Type":"ContainerStarted","Data":"56b617f3123afbe503d57efae6638c19e5cb65135255128afe92c3ba74b533d0"}
Sep 17 18:30:48.570130 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:48.570042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5nvz6" event={"ID":"0f609f65-9e6b-42fc-bc03-2809aed32e44","Type":"ContainerStarted","Data":"1c4368b7c91490db935bfc03d4198311e1b2ecb97e474ae5ff15f21ecdee3c2c"}
Sep 17 18:30:48.595201 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:48.595157    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5nvz6" podStartSLOduration=4.654780339 podStartE2EDuration="5.595143364s" totalImagesPullingTime="879.804533ms" totalInitContainerRuntime="60.558492ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:48.59453903 +0000 UTC m=+75.874196054" watchObservedRunningTime="2026-09-17 18:30:48.595143364 +0000 UTC m=+75.874800391"
Sep 17 18:30:48.776285 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:48.776245    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:48.876942 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:48.876866    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:49.216383 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:49.216291    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:49.325310 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:49.325277    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:49.415644 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:49.415619    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:50.013991 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:50.013957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.766072 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:53.766044    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.859911 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:53.859882    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.929858 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:53.929827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:54.890714 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:54.890666    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:54.975992 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:54.975950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:56.198726 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:30:56.198690    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:00.767472 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:00.767435    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:00.872995 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:00.872965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:00.940328 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:00.940294    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:02.141395 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:02.141365    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:02.379953 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:02.379921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:09.901943 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:09.901908    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:09.998776 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:09.998741    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:17.294135 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:17.294103    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6vfht_3aae0230-f1ff-48ca-b9b2-edf0dfe312ed/serve-healthcheck-canary/0.log"
Sep 17 18:31:17.567074 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:17.567045    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-szc74"
Sep 17 18:31:17.592260 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:17.592237    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:17.678874 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:17.678840    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:33.191332 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:33.191291    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:33.305268 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:33.305235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:33.402825 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:31:33.402796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:02.943801 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:32:02.943761    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:03.043782 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:32:03.043748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:11.250302 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:32:11.250266    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:11.318583 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:32:11.318550    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:14.526426 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:32:14.526381    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:22.357440 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:32:22.357351    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:22.454180 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:32:22.454150    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:47.434256 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:32:47.434221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:47.597172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:32:47.597135    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:07.185724 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:07.185685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:18.268424 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:18.268386    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:18.356304 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:18.356278    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:20.169882 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:20.169849    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:20.281314 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:20.281284    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:30.287255 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:30.287219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:30.373295 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:30.373265    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:30.553783 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:30.553756    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:30.669738 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:30.669704    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:30.861308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:30.861233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:39.897404 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:39.897366    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:39.995602 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:33:39.995572    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:32.631950 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:34:32.631918    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:34.156840 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:34:34.156811    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:34.279469 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:34:34.279440    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:34.429568 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:34:34.429490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:35.035619 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:34:35.035590    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:34:35.041075 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:34:35.041050    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:34:35.044308 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:34:35.044288    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:34:58.555160 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:34:58.555121    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:58.643274 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:34:58.643235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.632033 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:05.631998    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:05.808311 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:05.808281    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:06.088193 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.088158    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:06.186686 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.186635    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:06.262028 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.261997    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:06.493004 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.492929    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jsspz"]
Sep 17 18:35:06.495672 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.495657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jsspz"
Sep 17 18:35:06.498655 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.498630    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 18:35:06.499589 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.499565    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 18:35:06.499874 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.499857    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 18:35:06.500118 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.500103    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 18:35:06.500344 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.500329    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-mc7xb\""
Sep 17 18:35:06.502401 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.502372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:06.509805 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.509786    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jsspz"]
Sep 17 18:35:06.586545 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.586502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/39815e04-855e-4e3d-9497-dd2f31d232d7-certificates\") pod \"keda-admission-6c4d864b86-jsspz\" (UID: \"39815e04-855e-4e3d-9497-dd2f31d232d7\") " pod="openshift-keda/keda-admission-6c4d864b86-jsspz"
Sep 17 18:35:06.586545 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.586548    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcq7v\" (UniqueName: \"kubernetes.io/projected/39815e04-855e-4e3d-9497-dd2f31d232d7-kube-api-access-hcq7v\") pod \"keda-admission-6c4d864b86-jsspz\" (UID: \"39815e04-855e-4e3d-9497-dd2f31d232d7\") " pod="openshift-keda/keda-admission-6c4d864b86-jsspz"
Sep 17 18:35:06.687793 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.687758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/39815e04-855e-4e3d-9497-dd2f31d232d7-certificates\") pod \"keda-admission-6c4d864b86-jsspz\" (UID: \"39815e04-855e-4e3d-9497-dd2f31d232d7\") " pod="openshift-keda/keda-admission-6c4d864b86-jsspz"
Sep 17 18:35:06.688221 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.687804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcq7v\" (UniqueName: \"kubernetes.io/projected/39815e04-855e-4e3d-9497-dd2f31d232d7-kube-api-access-hcq7v\") pod \"keda-admission-6c4d864b86-jsspz\" (UID: \"39815e04-855e-4e3d-9497-dd2f31d232d7\") " pod="openshift-keda/keda-admission-6c4d864b86-jsspz"
Sep 17 18:35:06.690361 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.690339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/39815e04-855e-4e3d-9497-dd2f31d232d7-certificates\") pod \"keda-admission-6c4d864b86-jsspz\" (UID: \"39815e04-855e-4e3d-9497-dd2f31d232d7\") " pod="openshift-keda/keda-admission-6c4d864b86-jsspz"
Sep 17 18:35:06.697323 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.697299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcq7v\" (UniqueName: \"kubernetes.io/projected/39815e04-855e-4e3d-9497-dd2f31d232d7-kube-api-access-hcq7v\") pod \"keda-admission-6c4d864b86-jsspz\" (UID: \"39815e04-855e-4e3d-9497-dd2f31d232d7\") " pod="openshift-keda/keda-admission-6c4d864b86-jsspz"
Sep 17 18:35:06.774183 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.774075    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:06.809263 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:06.809227    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jsspz"
Sep 17 18:35:07.062215 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:07.062152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:07.100203 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:07.100181    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jsspz"]
Sep 17 18:35:07.102271 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:35:07.102244    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39815e04_855e_4e3d_9497_dd2f31d232d7.slice/crio-e97faa168bed72118b691e0f57df8934014a20cd6217a0a6042322fd378792dc WatchSource:0}: Error finding container e97faa168bed72118b691e0f57df8934014a20cd6217a0a6042322fd378792dc: Status 404 returned error can't find the container with id e97faa168bed72118b691e0f57df8934014a20cd6217a0a6042322fd378792dc
Sep 17 18:35:07.103915 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:07.103897    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:35:07.383799 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:07.383706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jsspz" event={"ID":"39815e04-855e-4e3d-9497-dd2f31d232d7","Type":"ContainerStarted","Data":"e97faa168bed72118b691e0f57df8934014a20cd6217a0a6042322fd378792dc"}
Sep 17 18:35:07.395183 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:07.395137    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:07.536309 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:07.536279    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:07.679940 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:07.679864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:10.396617 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:10.396577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jsspz" event={"ID":"39815e04-855e-4e3d-9497-dd2f31d232d7","Type":"ContainerStarted","Data":"2c3236eaaf8086572ecf9d306e41eb51241d51d22b634801e4e82dca9e44fc84"}
Sep 17 18:35:10.396973 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:10.396662    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-jsspz"
Sep 17 18:35:10.422451 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:10.422408    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-jsspz" podStartSLOduration=1.734500808 podStartE2EDuration="4.422393151s" totalImagesPullingTime="2.687892343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:10.422266334 +0000 UTC m=+337.701923363" watchObservedRunningTime="2026-09-17 18:35:10.422393151 +0000 UTC m=+337.702050223"
Sep 17 18:35:31.402396 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:31.402368    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-jsspz"
Sep 17 18:35:37.813468 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:37.813431    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:37.878473 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:37.878446    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:37.949877 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:37.949844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:38.479522 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:38.479470    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:38.581029 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:38.580982    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:49.145513 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:49.145462    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:49.243517 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:49.243458    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:49.315168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:35:49.315140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:00.732712 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:00.732671    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:00.832757 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:00.832724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:29.810551 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:29.810459    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:29.933197 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:29.933165    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"]
Sep 17 18:36:29.935425 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:29.935409    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:36:29.939300 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:36:29.939268    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-74.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-138-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 17 18:36:29.939456 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:36:29.939342    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-74.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-138-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 17 18:36:29.939456 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:36:29.939364    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisvc-webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-138-74.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-138-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" type="*v1.Secret"
Sep 17 18:36:29.939456 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:36:29.939376    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisvc-controller-manager-dockercfg-z8vnv\" is forbidden: User \"system:node:ip-10-0-138-74.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-138-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-z8vnv\"" type="*v1.Secret"
Sep 17 18:36:29.939658 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:36:29.939394    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"llmisvc-controller-manager-645f9c8fd6-zbf56\" is forbidden: User \"system:node:ip-10-0-138-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-138-74.ec2.internal' and this object" podUID="d37fab35-3cd3-479a-a6a4-a6e4e62459da" pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:36:29.955034 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:29.955013    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"]
Sep 17 18:36:29.965687 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:29.965667    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:30.039560 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:30.039531    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d37fab35-3cd3-479a-a6a4-a6e4e62459da-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-zbf56\" (UID: \"d37fab35-3cd3-479a-a6a4-a6e4e62459da\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:36:30.039679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:30.039584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68mm\" (UniqueName: \"kubernetes.io/projected/d37fab35-3cd3-479a-a6a4-a6e4e62459da-kube-api-access-z68mm\") pod \"llmisvc-controller-manager-645f9c8fd6-zbf56\" (UID: \"d37fab35-3cd3-479a-a6a4-a6e4e62459da\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:36:30.065788 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:30.065712    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:30.140803 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:30.140768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68mm\" (UniqueName: \"kubernetes.io/projected/d37fab35-3cd3-479a-a6a4-a6e4e62459da-kube-api-access-z68mm\") pod \"llmisvc-controller-manager-645f9c8fd6-zbf56\" (UID: \"d37fab35-3cd3-479a-a6a4-a6e4e62459da\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:36:30.140913 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:30.140826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d37fab35-3cd3-479a-a6a4-a6e4e62459da-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-zbf56\" (UID: \"d37fab35-3cd3-479a-a6a4-a6e4e62459da\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:36:30.160196 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:30.160170    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:30.766899 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:30.766864    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 18:36:30.769513 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:30.769468    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:30.773641 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:30.773625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d37fab35-3cd3-479a-a6a4-a6e4e62459da-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-zbf56\" (UID: \"d37fab35-3cd3-479a-a6a4-a6e4e62459da\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:36:30.820963 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:30.820933    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:36:31.321529 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:31.321501    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:36:31.324939 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:31.324912    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68mm\" (UniqueName: \"kubernetes.io/projected/d37fab35-3cd3-479a-a6a4-a6e4e62459da-kube-api-access-z68mm\") pod \"llmisvc-controller-manager-645f9c8fd6-zbf56\" (UID: \"d37fab35-3cd3-479a-a6a4-a6e4e62459da\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:36:31.343188 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:31.343132    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:31.422013 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:31.421988    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-z8vnv\""
Sep 17 18:36:31.447294 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:31.447269    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:31.449193 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:31.449179    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:36:31.739035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:31.738956    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"]
Sep 17 18:36:31.744673 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:36:31.744649    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd37fab35_3cd3_479a_a6a4_a6e4e62459da.slice/crio-c4a695ee7a748706ca42f8a0aa1d851d940a6ee3c090a72011b089d23decf177 WatchSource:0}: Error finding container c4a695ee7a748706ca42f8a0aa1d851d940a6ee3c090a72011b089d23decf177: Status 404 returned error can't find the container with id c4a695ee7a748706ca42f8a0aa1d851d940a6ee3c090a72011b089d23decf177
Sep 17 18:36:32.667343 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:32.667285    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56" event={"ID":"d37fab35-3cd3-479a-a6a4-a6e4e62459da","Type":"ContainerStarted","Data":"c4a695ee7a748706ca42f8a0aa1d851d940a6ee3c090a72011b089d23decf177"}
Sep 17 18:36:34.675973 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:34.675939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56" event={"ID":"d37fab35-3cd3-479a-a6a4-a6e4e62459da","Type":"ContainerStarted","Data":"56b397711ea6a372c41edf2e57d92929be8dd321fff81d3c38d2b30e6a8e058e"}
Sep 17 18:36:34.676367 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:34.676074    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:36:34.707092 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:36:34.707044    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56" podStartSLOduration=3.387993687 podStartE2EDuration="5.707027171s" totalImagesPullingTime="2.319033484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:34.706406602 +0000 UTC m=+421.986063630" watchObservedRunningTime="2026-09-17 18:36:34.707027171 +0000 UTC m=+421.986684202"
Sep 17 18:37:05.681340 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:05.681269    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-zbf56"
Sep 17 18:37:08.228108 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:08.228080    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:48.652415 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.652370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:48.689613 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.689582    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-pxkcr"]
Sep 17 18:37:48.692613 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.692595    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-pxkcr"
Sep 17 18:37:48.695491 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.695453    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-rj79z\""
Sep 17 18:37:48.695587 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.695546    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 18:37:48.702019 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.701994    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-pxkcr"]
Sep 17 18:37:48.737225 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.737190    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:48.762813 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.762610    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40266cf5-6bf8-4472-a725-af8ab4ffd67c-tls-certs\") pod \"model-serving-api-7b954ffc95-pxkcr\" (UID: \"40266cf5-6bf8-4472-a725-af8ab4ffd67c\") " pod="kserve/model-serving-api-7b954ffc95-pxkcr"
Sep 17 18:37:48.762813 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.762752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwtt\" (UniqueName: \"kubernetes.io/projected/40266cf5-6bf8-4472-a725-af8ab4ffd67c-kube-api-access-svwtt\") pod \"model-serving-api-7b954ffc95-pxkcr\" (UID: \"40266cf5-6bf8-4472-a725-af8ab4ffd67c\") " pod="kserve/model-serving-api-7b954ffc95-pxkcr"
Sep 17 18:37:48.833111 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.833078    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:48.863279 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.863245    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwtt\" (UniqueName: \"kubernetes.io/projected/40266cf5-6bf8-4472-a725-af8ab4ffd67c-kube-api-access-svwtt\") pod \"model-serving-api-7b954ffc95-pxkcr\" (UID: \"40266cf5-6bf8-4472-a725-af8ab4ffd67c\") " pod="kserve/model-serving-api-7b954ffc95-pxkcr"
Sep 17 18:37:48.863460 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.863315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40266cf5-6bf8-4472-a725-af8ab4ffd67c-tls-certs\") pod \"model-serving-api-7b954ffc95-pxkcr\" (UID: \"40266cf5-6bf8-4472-a725-af8ab4ffd67c\") " pod="kserve/model-serving-api-7b954ffc95-pxkcr"
Sep 17 18:37:48.865739 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.865716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40266cf5-6bf8-4472-a725-af8ab4ffd67c-tls-certs\") pod \"model-serving-api-7b954ffc95-pxkcr\" (UID: \"40266cf5-6bf8-4472-a725-af8ab4ffd67c\") " pod="kserve/model-serving-api-7b954ffc95-pxkcr"
Sep 17 18:37:48.872014 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.871994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwtt\" (UniqueName: \"kubernetes.io/projected/40266cf5-6bf8-4472-a725-af8ab4ffd67c-kube-api-access-svwtt\") pod \"model-serving-api-7b954ffc95-pxkcr\" (UID: \"40266cf5-6bf8-4472-a725-af8ab4ffd67c\") " pod="kserve/model-serving-api-7b954ffc95-pxkcr"
Sep 17 18:37:48.995278 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:48.995196    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:49.002810 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:49.002785    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-pxkcr"
Sep 17 18:37:49.151742 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:49.151707    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:49.275875 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:49.275802    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-pxkcr"]
Sep 17 18:37:49.279468 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:37:49.279442    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40266cf5_6bf8_4472_a725_af8ab4ffd67c.slice/crio-eb6e15a1579441d8b4a4509a38a9b2c1594b632fba0cd97f55ed2ab209e2a460 WatchSource:0}: Error finding container eb6e15a1579441d8b4a4509a38a9b2c1594b632fba0cd97f55ed2ab209e2a460: Status 404 returned error can't find the container with id eb6e15a1579441d8b4a4509a38a9b2c1594b632fba0cd97f55ed2ab209e2a460
Sep 17 18:37:49.928081 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:49.928039    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-pxkcr" event={"ID":"40266cf5-6bf8-4472-a725-af8ab4ffd67c","Type":"ContainerStarted","Data":"eb6e15a1579441d8b4a4509a38a9b2c1594b632fba0cd97f55ed2ab209e2a460"}
Sep 17 18:37:49.955332 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:49.955295    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:50.055769 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:50.055737    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:51.936929 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:51.936897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-pxkcr" event={"ID":"40266cf5-6bf8-4472-a725-af8ab4ffd67c","Type":"ContainerStarted","Data":"e4196542a12b49d9cfb82764c3659571f3163d767d0964a73f9f6734f12fc102"}
Sep 17 18:37:51.937349 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:51.937011    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-pxkcr"
Sep 17 18:37:51.955659 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:37:51.955616    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-pxkcr" podStartSLOduration=1.822367493 podStartE2EDuration="3.955600455s" totalImagesPullingTime="2.133232962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:51.953454181 +0000 UTC m=+499.233111209" watchObservedRunningTime="2026-09-17 18:37:51.955600455 +0000 UTC m=+499.235257484"
Sep 17 18:38:02.943848 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:38:02.943816    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-pxkcr"
Sep 17 18:39:20.215344 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:20.215306    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:23.055237 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:23.055196    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:23.155097 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:23.155068    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:34.604310 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:34.604273    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:35.059603 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:35.059574    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:39:35.060459 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:35.060435    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:39:35.730351 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:35.730312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:35.827044 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:35.827015    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:35.944029 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:35.944000    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:36.042183 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.042109    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"]
Sep 17 18:39:36.045260 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.045244    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.047810 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.047791    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:39:36.048733 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.048710    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:39:36.048813 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.048779    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:39:36.048813 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.048790    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\""
Sep 17 18:39:36.048919 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.048826    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:39:36.048919 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.048780    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:39:36.049026 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.048992    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:39:36.059428 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.059408    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"]
Sep 17 18:39:36.089523 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.089497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.089657 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.089529    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wckt9\" (UniqueName: \"kubernetes.io/projected/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-api-access-wckt9\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.089657 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.089552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bea83938-82f1-4e50-ab51-b9c65e71d6d5-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.089657 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.089570    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.089754 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.089651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.189980 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.189942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.190172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.189995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.190172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.190024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wckt9\" (UniqueName: \"kubernetes.io/projected/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-api-access-wckt9\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.190172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.190052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bea83938-82f1-4e50-ab51-b9c65e71d6d5-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.190172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.190077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.190395 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:39:36.190170    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:39:36.190395 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:39:36.190245    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea83938-82f1-4e50-ab51-b9c65e71d6d5-proxy-tls podName:bea83938-82f1-4e50-ab51-b9c65e71d6d5 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:36.690223969 +0000 UTC m=+603.969880978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bea83938-82f1-4e50-ab51-b9c65e71d6d5-proxy-tls") pod "isvc-canary-rollback-predictor-7679bbb844-ltkln" (UID: "bea83938-82f1-4e50-ab51-b9c65e71d6d5") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:39:36.190538 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.190504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.190753 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.190734    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.190814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.190736    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.201866 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.201843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wckt9\" (UniqueName: \"kubernetes.io/projected/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-api-access-wckt9\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.302203 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.302174    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:36.692961 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.692860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bea83938-82f1-4e50-ab51-b9c65e71d6d5-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.695262 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.695232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bea83938-82f1-4e50-ab51-b9c65e71d6d5-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-ltkln\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:36.955448 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:36.955363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:39:37.248160 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:37.248079    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"]
Sep 17 18:39:37.252292 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:39:37.252256    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea83938_82f1_4e50_ab51_b9c65e71d6d5.slice/crio-c4ee12ec375d65027cd5537d150ab3a833c9fc152d444e530f89fb4713e3fe08 WatchSource:0}: Error finding container c4ee12ec375d65027cd5537d150ab3a833c9fc152d444e530f89fb4713e3fe08: Status 404 returned error can't find the container with id c4ee12ec375d65027cd5537d150ab3a833c9fc152d444e530f89fb4713e3fe08
Sep 17 18:39:37.283436 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:37.283397    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" event={"ID":"bea83938-82f1-4e50-ab51-b9c65e71d6d5","Type":"ContainerStarted","Data":"c4ee12ec375d65027cd5537d150ab3a833c9fc152d444e530f89fb4713e3fe08"}
Sep 17 18:39:40.296904 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:40.296863    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" event={"ID":"bea83938-82f1-4e50-ab51-b9c65e71d6d5","Type":"ContainerStarted","Data":"4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a"}
Sep 17 18:39:41.161731 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:41.161696    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:44.312197 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:44.312158    2561 generic.go:356] "Generic (PLEG): container finished" podID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerID="4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a" exitCode=0
Sep 17 18:39:44.312584 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:44.312230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" event={"ID":"bea83938-82f1-4e50-ab51-b9c65e71d6d5","Type":"ContainerDied","Data":"4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a"}
Sep 17 18:39:51.658554 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:51.658443    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:57.374410 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:39:57.374321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" event={"ID":"bea83938-82f1-4e50-ab51-b9c65e71d6d5","Type":"ContainerStarted","Data":"c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4"}
Sep 17 18:40:00.388517 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:00.388465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" event={"ID":"bea83938-82f1-4e50-ab51-b9c65e71d6d5","Type":"ContainerStarted","Data":"ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c"}
Sep 17 18:40:00.388917 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:00.388644    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:40:00.411854 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:00.411756    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podStartSLOduration=2.7188597420000002 podStartE2EDuration="24.411718738s" totalImagesPullingTime="17.942569944s" totalInitContainerRuntime="3.750289052s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:00.411703461 +0000 UTC m=+627.691360503" watchObservedRunningTime="2026-09-17 18:40:00.411718738 +0000 UTC m=+627.691375768"
Sep 17 18:40:01.391911 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:01.391881    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:40:01.393215 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:01.393181    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 17 18:40:02.394826 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:02.394787    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 17 18:40:07.399885 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:07.399846    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:40:07.400463 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:07.400433    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 17 18:40:12.964213 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:12.964152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:15.815736 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:15.815700    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:17.400359 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:17.400312    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 17 18:40:27.400794 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:27.400750    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 17 18:40:37.400828 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:37.400785    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 17 18:40:47.400984 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:47.400938    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 17 18:40:57.400633 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:40:57.400592    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 17 18:41:07.400903 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:07.400860    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.15:8080: connect: connection refused"
Sep 17 18:41:11.217990 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:11.217961    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:41:25.638814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:25.638784    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:30.665798 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:30.665760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:30.793290 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:30.793256    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:30.898011 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:30.897985    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:30.932008 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:30.931941    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"]
Sep 17 18:41:30.934243 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:30.934226    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:30.936610 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:30.936587    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e4246-predictor-serving-cert\""
Sep 17 18:41:30.936703 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:30.936592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e4246-kube-rbac-proxy-sar-config\""
Sep 17 18:41:30.947128 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:30.947109    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"]
Sep 17 18:41:31.030952 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.030920    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/685172c6-c9b8-40d9-a705-2c613703399c-proxy-tls\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.031138 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.030957    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.031138 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.031008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.031138 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.031037    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcnl6\" (UniqueName: \"kubernetes.io/projected/685172c6-c9b8-40d9-a705-2c613703399c-kube-api-access-dcnl6\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.031138 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.031063    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/685172c6-c9b8-40d9-a705-2c613703399c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.046247 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.046219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:31.132206 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.132168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.132375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.132220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcnl6\" (UniqueName: \"kubernetes.io/projected/685172c6-c9b8-40d9-a705-2c613703399c-kube-api-access-dcnl6\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.132375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.132297    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/685172c6-c9b8-40d9-a705-2c613703399c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.132375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.132344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/685172c6-c9b8-40d9-a705-2c613703399c-proxy-tls\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.132582 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.132381    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.132778 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.132755    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/685172c6-c9b8-40d9-a705-2c613703399c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.133068 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.133042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.133174 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.133048    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.134834 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.134813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/685172c6-c9b8-40d9-a705-2c613703399c-proxy-tls\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.141456 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.141435    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcnl6\" (UniqueName: \"kubernetes.io/projected/685172c6-c9b8-40d9-a705-2c613703399c-kube-api-access-dcnl6\") pod \"isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.241517 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.241427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:31.244406 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.244387    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:31.518185 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.518140    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"]
Sep 17 18:41:31.520658 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:41:31.520632    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685172c6_c9b8_40d9_a705_2c613703399c.slice/crio-7fab6aae20125f8aebbcd45857606cb66db7a05b7d59f3d7caac09fc8ec89396 WatchSource:0}: Error finding container 7fab6aae20125f8aebbcd45857606cb66db7a05b7d59f3d7caac09fc8ec89396: Status 404 returned error can't find the container with id 7fab6aae20125f8aebbcd45857606cb66db7a05b7d59f3d7caac09fc8ec89396
Sep 17 18:41:31.703214 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.703173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" event={"ID":"685172c6-c9b8-40d9-a705-2c613703399c","Type":"ContainerStarted","Data":"2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96"}
Sep 17 18:41:31.703693 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:31.703230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" event={"ID":"685172c6-c9b8-40d9-a705-2c613703399c","Type":"ContainerStarted","Data":"7fab6aae20125f8aebbcd45857606cb66db7a05b7d59f3d7caac09fc8ec89396"}
Sep 17 18:41:35.718893 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:35.718863    2561 generic.go:356] "Generic (PLEG): container finished" podID="685172c6-c9b8-40d9-a705-2c613703399c" containerID="2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96" exitCode=0
Sep 17 18:41:35.719270 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:35.718912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" event={"ID":"685172c6-c9b8-40d9-a705-2c613703399c","Type":"ContainerDied","Data":"2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96"}
Sep 17 18:41:35.720125 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:35.720109    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:41:56.814290 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:56.814253    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" event={"ID":"685172c6-c9b8-40d9-a705-2c613703399c","Type":"ContainerStarted","Data":"e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19"}
Sep 17 18:41:56.814778 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:56.814358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" event={"ID":"685172c6-c9b8-40d9-a705-2c613703399c","Type":"ContainerStarted","Data":"b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771"}
Sep 17 18:41:56.814778 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:56.814596    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:56.837176 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:56.837083    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podStartSLOduration=2.846664513 podStartE2EDuration="26.837065221s" totalImagesPullingTime="20.327665475s" totalInitContainerRuntime="3.662735233s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:56.835888158 +0000 UTC m=+744.115545187" watchObservedRunningTime="2026-09-17 18:41:56.837065221 +0000 UTC m=+744.116722249"
Sep 17 18:41:57.817955 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:57.817917    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:41:57.819217 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:57.819191    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 17 18:41:58.821370 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:41:58.821322    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 17 18:42:03.825565 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:03.825533    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:42:03.826125 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:03.826075    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 17 18:42:13.826683 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:13.826643    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 17 18:42:23.826684 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:23.826642    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 17 18:42:33.826886 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:33.826851    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 17 18:42:43.826261 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:43.826223    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 17 18:42:53.826152 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:53.826067    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 17 18:42:55.724230 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:55.724194    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"]
Sep 17 18:42:55.724659 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:55.724549    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container" containerID="cri-o://c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4" gracePeriod=30
Sep 17 18:42:55.724659 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:55.724600    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kube-rbac-proxy" containerID="cri-o://ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c" gracePeriod=30
Sep 17 18:42:55.755251 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:55.755220    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:56.034194 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:56.034110    2561 generic.go:356] "Generic (PLEG): container finished" podID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerID="ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c" exitCode=2
Sep 17 18:42:56.034194 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:56.034157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" event={"ID":"bea83938-82f1-4e50-ab51-b9c65e71d6d5","Type":"ContainerDied","Data":"ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c"}
Sep 17 18:42:57.395822 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:42:57.395780    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.15:8643/healthz\": dial tcp 10.134.0.15:8643: connect: connection refused"
Sep 17 18:43:00.463232 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.463208    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:43:00.599088 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.599056    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bea83938-82f1-4e50-ab51-b9c65e71d6d5-proxy-tls\") pod \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") "
Sep 17 18:43:00.599248 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.599123    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-cabundle-cert\") pod \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") "
Sep 17 18:43:00.599248 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.599149    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wckt9\" (UniqueName: \"kubernetes.io/projected/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-api-access-wckt9\") pod \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") "
Sep 17 18:43:00.599248 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.599178    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-rbac-proxy-sar-config\") pod \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") "
Sep 17 18:43:00.599248 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.599211    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kserve-provision-location\") pod \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\" (UID: \"bea83938-82f1-4e50-ab51-b9c65e71d6d5\") "
Sep 17 18:43:00.599577 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.599540    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bea83938-82f1-4e50-ab51-b9c65e71d6d5" (UID: "bea83938-82f1-4e50-ab51-b9c65e71d6d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:00.599688 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.599580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bea83938-82f1-4e50-ab51-b9c65e71d6d5" (UID: "bea83938-82f1-4e50-ab51-b9c65e71d6d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:00.599688 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.599604    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bea83938-82f1-4e50-ab51-b9c65e71d6d5" (UID: "bea83938-82f1-4e50-ab51-b9c65e71d6d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:00.601400 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.601367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-api-access-wckt9" (OuterVolumeSpecName: "kube-api-access-wckt9") pod "bea83938-82f1-4e50-ab51-b9c65e71d6d5" (UID: "bea83938-82f1-4e50-ab51-b9c65e71d6d5"). InnerVolumeSpecName "kube-api-access-wckt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:00.601493 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.601374    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bea83938-82f1-4e50-ab51-b9c65e71d6d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bea83938-82f1-4e50-ab51-b9c65e71d6d5" (UID: "bea83938-82f1-4e50-ab51-b9c65e71d6d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:00.699787 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.699751    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.699787 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.699780    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bea83938-82f1-4e50-ab51-b9c65e71d6d5-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.699787 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.699792    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.700013 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.699800    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wckt9\" (UniqueName: \"kubernetes.io/projected/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-api-access-wckt9\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.700013 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.699808    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bea83938-82f1-4e50-ab51-b9c65e71d6d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.906724 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.906651    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:00.989629 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.989595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:00.994800 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.994775    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"]
Sep 17 18:43:00.995052 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.995041    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="storage-initializer"
Sep 17 18:43:00.995097 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.995054    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="storage-initializer"
Sep 17 18:43:00.995097 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.995063    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kube-rbac-proxy"
Sep 17 18:43:00.995097 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.995069    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kube-rbac-proxy"
Sep 17 18:43:00.995097 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.995082    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container"
Sep 17 18:43:00.995097 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.995088    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container"
Sep 17 18:43:00.995237 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.995128    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kube-rbac-proxy"
Sep 17 18:43:00.995237 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.995137    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerName="kserve-container"
Sep 17 18:43:00.997172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.997158    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:00.999918 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:00.999897    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:43:01.000347 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.000332    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:43:01.013607 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.013587    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"]
Sep 17 18:43:01.053038 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.053015    2561 generic.go:356] "Generic (PLEG): container finished" podID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" containerID="c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4" exitCode=0
Sep 17 18:43:01.053151 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.053066    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"
Sep 17 18:43:01.053151 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.053087    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" event={"ID":"bea83938-82f1-4e50-ab51-b9c65e71d6d5","Type":"ContainerDied","Data":"c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4"}
Sep 17 18:43:01.053219 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.053170    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln" event={"ID":"bea83938-82f1-4e50-ab51-b9c65e71d6d5","Type":"ContainerDied","Data":"c4ee12ec375d65027cd5537d150ab3a833c9fc152d444e530f89fb4713e3fe08"}
Sep 17 18:43:01.053219 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.053197    2561 scope.go:117] "RemoveContainer" containerID="ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c"
Sep 17 18:43:01.061770 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.061754    2561 scope.go:117] "RemoveContainer" containerID="c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4"
Sep 17 18:43:01.068617 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.068600    2561 scope.go:117] "RemoveContainer" containerID="4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a"
Sep 17 18:43:01.075210 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.075193    2561 scope.go:117] "RemoveContainer" containerID="ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c"
Sep 17 18:43:01.075492 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:43:01.075459    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c\": container with ID starting with ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c not found: ID does not exist" containerID="ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c"
Sep 17 18:43:01.075571 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.075510    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c"} err="failed to get container status \"ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c\": rpc error: code = NotFound desc = could not find container \"ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c\": container with ID starting with ef35d569d4d7c4a161b6529f61da4e13e678bc1c111703fcb7639bbab248fb1c not found: ID does not exist"
Sep 17 18:43:01.075571 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.075551    2561 scope.go:117] "RemoveContainer" containerID="c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4"
Sep 17 18:43:01.075848 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:43:01.075825    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4\": container with ID starting with c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4 not found: ID does not exist" containerID="c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4"
Sep 17 18:43:01.075918 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.075855    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4"} err="failed to get container status \"c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4\": rpc error: code = NotFound desc = could not find container \"c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4\": container with ID starting with c12d2e10653575048159bb2cc03ff1c108de95bb851fe348a48b617d3e234fe4 not found: ID does not exist"
Sep 17 18:43:01.075918 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.075871    2561 scope.go:117] "RemoveContainer" containerID="4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a"
Sep 17 18:43:01.076164 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:43:01.076142    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a\": container with ID starting with 4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a not found: ID does not exist" containerID="4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a"
Sep 17 18:43:01.076222 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.076174    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a"} err="failed to get container status \"4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a\": rpc error: code = NotFound desc = could not find container \"4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a\": container with ID starting with 4b2740eeb6c1d7d2f28a36b6d3b71862c8c787e425413115db9d8bc49c28c16a not found: ID does not exist"
Sep 17 18:43:01.077717 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.077699    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"]
Sep 17 18:43:01.079601 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.079584    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-ltkln"]
Sep 17 18:43:01.103700 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.103678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61bfd0f-285b-420b-8dac-47759689524c-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.103794 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.103712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.103794 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.103745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.103892 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.103818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lwc\" (UniqueName: \"kubernetes.io/projected/e61bfd0f-285b-420b-8dac-47759689524c-kube-api-access-98lwc\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.103892 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.103872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e61bfd0f-285b-420b-8dac-47759689524c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.205008 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.204912    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lwc\" (UniqueName: \"kubernetes.io/projected/e61bfd0f-285b-420b-8dac-47759689524c-kube-api-access-98lwc\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.205008 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.204956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e61bfd0f-285b-420b-8dac-47759689524c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.205008 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.205000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61bfd0f-285b-420b-8dac-47759689524c-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.205264 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.205019    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.205264 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.205043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.205264 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:43:01.205147    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:43:01.205264 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:43:01.205213    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e61bfd0f-285b-420b-8dac-47759689524c-proxy-tls podName:e61bfd0f-285b-420b-8dac-47759689524c nodeName:}" failed. No retries permitted until 2026-09-17 18:43:01.705193418 +0000 UTC m=+808.984850426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e61bfd0f-285b-420b-8dac-47759689524c-proxy-tls") pod "isvc-canary-stop-predictor-7968cbc88d-clvgb" (UID: "e61bfd0f-285b-420b-8dac-47759689524c") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:43:01.205532 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.205514    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e61bfd0f-285b-420b-8dac-47759689524c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.205843 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.205821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.205881 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.205830    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.217653 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.217630    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bea83938-82f1-4e50-ab51-b9c65e71d6d5" path="/var/lib/kubelet/pods/bea83938-82f1-4e50-ab51-b9c65e71d6d5/volumes"
Sep 17 18:43:01.218403 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.218387    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lwc\" (UniqueName: \"kubernetes.io/projected/e61bfd0f-285b-420b-8dac-47759689524c-kube-api-access-98lwc\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.561625 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.561598    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:01.662051 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.662015    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:01.709638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.709607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61bfd0f-285b-420b-8dac-47759689524c-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.711937 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.711908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61bfd0f-285b-420b-8dac-47759689524c-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-clvgb\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:01.907417 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:01.907304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:02.189547 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:02.189519    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"]
Sep 17 18:43:02.190943 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:43:02.190910    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61bfd0f_285b_420b_8dac_47759689524c.slice/crio-230bf6eb3916e4780a4ce82b0709bf4d3f99d86d32a7ace8445f994cb6eaf44d WatchSource:0}: Error finding container 230bf6eb3916e4780a4ce82b0709bf4d3f99d86d32a7ace8445f994cb6eaf44d: Status 404 returned error can't find the container with id 230bf6eb3916e4780a4ce82b0709bf4d3f99d86d32a7ace8445f994cb6eaf44d
Sep 17 18:43:03.062382 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:03.062346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" event={"ID":"e61bfd0f-285b-420b-8dac-47759689524c","Type":"ContainerStarted","Data":"6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1"}
Sep 17 18:43:03.062786 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:03.062405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" event={"ID":"e61bfd0f-285b-420b-8dac-47759689524c","Type":"ContainerStarted","Data":"230bf6eb3916e4780a4ce82b0709bf4d3f99d86d32a7ace8445f994cb6eaf44d"}
Sep 17 18:43:03.826658 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:03.826626    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:43:06.074019 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:06.073991    2561 generic.go:356] "Generic (PLEG): container finished" podID="e61bfd0f-285b-420b-8dac-47759689524c" containerID="6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1" exitCode=0
Sep 17 18:43:06.074378 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:06.074062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" event={"ID":"e61bfd0f-285b-420b-8dac-47759689524c","Type":"ContainerDied","Data":"6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1"}
Sep 17 18:43:07.079942 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:07.079907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" event={"ID":"e61bfd0f-285b-420b-8dac-47759689524c","Type":"ContainerStarted","Data":"7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8"}
Sep 17 18:43:07.080337 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:07.079992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" event={"ID":"e61bfd0f-285b-420b-8dac-47759689524c","Type":"ContainerStarted","Data":"0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8"}
Sep 17 18:43:07.080337 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:07.080228    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:07.105222 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:07.105171    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podStartSLOduration=3.369204927 podStartE2EDuration="7.105156624s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.735951697s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:07.103185274 +0000 UTC m=+814.382842304" watchObservedRunningTime="2026-09-17 18:43:07.105156624 +0000 UTC m=+814.384813652"
Sep 17 18:43:08.083204 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:08.083168    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:08.084337 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:08.084311    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 17 18:43:09.087168 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:09.087123    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 17 18:43:10.841689 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:10.841657    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"]
Sep 17 18:43:10.843715 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:10.843701    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:10.846261 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:10.846238    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-e4246-serving-cert\""
Sep 17 18:43:10.846371 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:10.846237    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-e4246-kube-rbac-proxy-sar-config\""
Sep 17 18:43:10.847586 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:10.847563    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:10.855191 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:10.855169    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"]
Sep 17 18:43:10.978131 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:10.978098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc6f5827-4d2b-410b-96fe-5b912c7c6177-openshift-service-ca-bundle\") pod \"model-chainer-raw-e4246-59f9c894c4-ggxrj\" (UID: \"fc6f5827-4d2b-410b-96fe-5b912c7c6177\") " pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:10.978284 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:10.978159    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f5827-4d2b-410b-96fe-5b912c7c6177-proxy-tls\") pod \"model-chainer-raw-e4246-59f9c894c4-ggxrj\" (UID: \"fc6f5827-4d2b-410b-96fe-5b912c7c6177\") " pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:11.079215 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:11.079181    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f5827-4d2b-410b-96fe-5b912c7c6177-proxy-tls\") pod \"model-chainer-raw-e4246-59f9c894c4-ggxrj\" (UID: \"fc6f5827-4d2b-410b-96fe-5b912c7c6177\") " pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:11.079351 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:11.079240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc6f5827-4d2b-410b-96fe-5b912c7c6177-openshift-service-ca-bundle\") pod \"model-chainer-raw-e4246-59f9c894c4-ggxrj\" (UID: \"fc6f5827-4d2b-410b-96fe-5b912c7c6177\") " pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:11.079900 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:11.079871    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc6f5827-4d2b-410b-96fe-5b912c7c6177-openshift-service-ca-bundle\") pod \"model-chainer-raw-e4246-59f9c894c4-ggxrj\" (UID: \"fc6f5827-4d2b-410b-96fe-5b912c7c6177\") " pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:11.081694 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:11.081672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f5827-4d2b-410b-96fe-5b912c7c6177-proxy-tls\") pod \"model-chainer-raw-e4246-59f9c894c4-ggxrj\" (UID: \"fc6f5827-4d2b-410b-96fe-5b912c7c6177\") " pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:11.106828 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:11.106764    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:11.153187 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:11.153162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:11.421595 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:11.421512    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"]
Sep 17 18:43:11.424495 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:43:11.424456    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6f5827_4d2b_410b_96fe_5b912c7c6177.slice/crio-75762f168f8580f71ef80140ab2011d23ce17aab1160c72883e4139e578aaf13 WatchSource:0}: Error finding container 75762f168f8580f71ef80140ab2011d23ce17aab1160c72883e4139e578aaf13: Status 404 returned error can't find the container with id 75762f168f8580f71ef80140ab2011d23ce17aab1160c72883e4139e578aaf13
Sep 17 18:43:12.098140 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:12.098104    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" event={"ID":"fc6f5827-4d2b-410b-96fe-5b912c7c6177","Type":"ContainerStarted","Data":"75762f168f8580f71ef80140ab2011d23ce17aab1160c72883e4139e578aaf13"}
Sep 17 18:43:14.091790 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:14.091762    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:43:14.092227 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:14.092203    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 17 18:43:14.106712 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:14.106682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" event={"ID":"fc6f5827-4d2b-410b-96fe-5b912c7c6177","Type":"ContainerStarted","Data":"f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab"}
Sep 17 18:43:14.106830 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:14.106737    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:14.126178 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:14.126128    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" podStartSLOduration=2.037837623 podStartE2EDuration="4.126113171s" totalImagesPullingTime="2.088275548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:14.125847869 +0000 UTC m=+821.405504901" watchObservedRunningTime="2026-09-17 18:43:14.126113171 +0000 UTC m=+821.405770200"
Sep 17 18:43:20.116418 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:20.116392    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:20.892496 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:20.892454    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"]
Sep 17 18:43:20.892679 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:20.892657    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerName="model-chainer-raw-e4246" containerID="cri-o://f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab" gracePeriod=30
Sep 17 18:43:20.928675 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:20.928644    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:21.064704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.064653    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:21.150059 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.149979    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"]
Sep 17 18:43:21.150502 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.150227    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container" containerID="cri-o://b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771" gracePeriod=30
Sep 17 18:43:21.150502 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.150239    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kube-rbac-proxy" containerID="cri-o://e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19" gracePeriod=30
Sep 17 18:43:21.156876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.156843    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:21.252662 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.252597    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:21.258190 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.258162    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"]
Sep 17 18:43:21.261472 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.261454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.264463 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.264442    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-7a45d-predictor-serving-cert\""
Sep 17 18:43:21.264565 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.264532    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-7a45d-kube-rbac-proxy-sar-config\""
Sep 17 18:43:21.279230 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.279211    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"]
Sep 17 18:43:21.354095 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.354067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.354264 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.354130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e96f3f-6976-4259-8bdc-3b69a4a90afc-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.354264 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.354172    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68zn\" (UniqueName: \"kubernetes.io/projected/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-api-access-x68zn\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.354264 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.354213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.354264 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.354236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.358885 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.358865    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:21.455271 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.455189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68zn\" (UniqueName: \"kubernetes.io/projected/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-api-access-x68zn\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.455271 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.455236    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.455271 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.455263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.455700 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.455293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.455700 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.455445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e96f3f-6976-4259-8bdc-3b69a4a90afc-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.455700 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.455680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.455975 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.455955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.456038 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.456004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.457795 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.457770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e96f3f-6976-4259-8bdc-3b69a4a90afc-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.463840 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.463813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68zn\" (UniqueName: \"kubernetes.io/projected/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-api-access-x68zn\") pod \"isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.524322 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.524297    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:21.570684 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.570656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:21.842322 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:21.842292    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"]
Sep 17 18:43:21.846091 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:43:21.846062    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e96f3f_6976_4259_8bdc_3b69a4a90afc.slice/crio-ab21bfa92b8577878ed58c0748efe9a3c9cbd9fef8ab9c80661af17a9111f64f WatchSource:0}: Error finding container ab21bfa92b8577878ed58c0748efe9a3c9cbd9fef8ab9c80661af17a9111f64f: Status 404 returned error can't find the container with id ab21bfa92b8577878ed58c0748efe9a3c9cbd9fef8ab9c80661af17a9111f64f
Sep 17 18:43:22.138459 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:22.138371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" event={"ID":"09e96f3f-6976-4259-8bdc-3b69a4a90afc","Type":"ContainerStarted","Data":"81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc"}
Sep 17 18:43:22.138459 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:22.138448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" event={"ID":"09e96f3f-6976-4259-8bdc-3b69a4a90afc","Type":"ContainerStarted","Data":"ab21bfa92b8577878ed58c0748efe9a3c9cbd9fef8ab9c80661af17a9111f64f"}
Sep 17 18:43:22.140704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:22.140674    2561 generic.go:356] "Generic (PLEG): container finished" podID="685172c6-c9b8-40d9-a705-2c613703399c" containerID="e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19" exitCode=2
Sep 17 18:43:22.140816 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:22.140727    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" event={"ID":"685172c6-c9b8-40d9-a705-2c613703399c","Type":"ContainerDied","Data":"e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19"}
Sep 17 18:43:23.821731 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:23.821693    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.16:8643/healthz\": dial tcp 10.134.0.16:8643: connect: connection refused"
Sep 17 18:43:23.825974 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:23.825951    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Sep 17 18:43:24.093033 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:24.092952    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 17 18:43:25.093729 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.093705    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:43:25.115279 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.115248    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerName="model-chainer-raw-e4246" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:43:25.154080 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.154050    2561 generic.go:356] "Generic (PLEG): container finished" podID="685172c6-c9b8-40d9-a705-2c613703399c" containerID="b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771" exitCode=0
Sep 17 18:43:25.154206 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.154102    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"
Sep 17 18:43:25.154206 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.154123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" event={"ID":"685172c6-c9b8-40d9-a705-2c613703399c","Type":"ContainerDied","Data":"b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771"}
Sep 17 18:43:25.154206 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.154198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw" event={"ID":"685172c6-c9b8-40d9-a705-2c613703399c","Type":"ContainerDied","Data":"7fab6aae20125f8aebbcd45857606cb66db7a05b7d59f3d7caac09fc8ec89396"}
Sep 17 18:43:25.154307 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.154227    2561 scope.go:117] "RemoveContainer" containerID="e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19"
Sep 17 18:43:25.162507 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.162415    2561 scope.go:117] "RemoveContainer" containerID="b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771"
Sep 17 18:43:25.170829 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.170813    2561 scope.go:117] "RemoveContainer" containerID="2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96"
Sep 17 18:43:25.177624 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.177605    2561 scope.go:117] "RemoveContainer" containerID="e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19"
Sep 17 18:43:25.177882 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:43:25.177864    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19\": container with ID starting with e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19 not found: ID does not exist" containerID="e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19"
Sep 17 18:43:25.177927 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.177890    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19"} err="failed to get container status \"e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19\": rpc error: code = NotFound desc = could not find container \"e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19\": container with ID starting with e36d16244107892e41d830bbe8beb7d0b452537444e6c9c5a97d30cf116d8d19 not found: ID does not exist"
Sep 17 18:43:25.177927 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.177906    2561 scope.go:117] "RemoveContainer" containerID="b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771"
Sep 17 18:43:25.178141 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:43:25.178124    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771\": container with ID starting with b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771 not found: ID does not exist" containerID="b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771"
Sep 17 18:43:25.178181 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.178147    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771"} err="failed to get container status \"b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771\": rpc error: code = NotFound desc = could not find container \"b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771\": container with ID starting with b4d135dded32472243b5d5fcb2eca41aa63dd9c39a6ff7f1e9602a485a7c3771 not found: ID does not exist"
Sep 17 18:43:25.178181 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.178162    2561 scope.go:117] "RemoveContainer" containerID="2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96"
Sep 17 18:43:25.178399 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:43:25.178384    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96\": container with ID starting with 2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96 not found: ID does not exist" containerID="2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96"
Sep 17 18:43:25.178444 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.178402    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96"} err="failed to get container status \"2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96\": rpc error: code = NotFound desc = could not find container \"2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96\": container with ID starting with 2edf66fefec9db4f1926dac1c7241a14407ef8865d2e3958aa6b1b8387c03a96 not found: ID does not exist"
Sep 17 18:43:25.285352 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.285271    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcnl6\" (UniqueName: \"kubernetes.io/projected/685172c6-c9b8-40d9-a705-2c613703399c-kube-api-access-dcnl6\") pod \"685172c6-c9b8-40d9-a705-2c613703399c\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") "
Sep 17 18:43:25.285352 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.285333    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-cabundle-cert\") pod \"685172c6-c9b8-40d9-a705-2c613703399c\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") "
Sep 17 18:43:25.285578 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.285403    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/685172c6-c9b8-40d9-a705-2c613703399c-proxy-tls\") pod \"685172c6-c9b8-40d9-a705-2c613703399c\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") "
Sep 17 18:43:25.285578 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.285427    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-kube-rbac-proxy-sar-config\") pod \"685172c6-c9b8-40d9-a705-2c613703399c\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") "
Sep 17 18:43:25.285578 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.285499    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/685172c6-c9b8-40d9-a705-2c613703399c-kserve-provision-location\") pod \"685172c6-c9b8-40d9-a705-2c613703399c\" (UID: \"685172c6-c9b8-40d9-a705-2c613703399c\") "
Sep 17 18:43:25.285854 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.285816    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685172c6-c9b8-40d9-a705-2c613703399c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "685172c6-c9b8-40d9-a705-2c613703399c" (UID: "685172c6-c9b8-40d9-a705-2c613703399c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:25.285988 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.285835    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "685172c6-c9b8-40d9-a705-2c613703399c" (UID: "685172c6-c9b8-40d9-a705-2c613703399c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:25.285988 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.285882    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "685172c6-c9b8-40d9-a705-2c613703399c" (UID: "685172c6-c9b8-40d9-a705-2c613703399c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:25.287432 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.287406    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685172c6-c9b8-40d9-a705-2c613703399c-kube-api-access-dcnl6" (OuterVolumeSpecName: "kube-api-access-dcnl6") pod "685172c6-c9b8-40d9-a705-2c613703399c" (UID: "685172c6-c9b8-40d9-a705-2c613703399c"). InnerVolumeSpecName "kube-api-access-dcnl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:25.287556 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.287458    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685172c6-c9b8-40d9-a705-2c613703399c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "685172c6-c9b8-40d9-a705-2c613703399c" (UID: "685172c6-c9b8-40d9-a705-2c613703399c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:25.387033 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.387002    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.387033 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.387028    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/685172c6-c9b8-40d9-a705-2c613703399c-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.387033 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.387041    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/685172c6-c9b8-40d9-a705-2c613703399c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.387264 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.387054    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/685172c6-c9b8-40d9-a705-2c613703399c-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.387264 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.387064    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcnl6\" (UniqueName: \"kubernetes.io/projected/685172c6-c9b8-40d9-a705-2c613703399c-kube-api-access-dcnl6\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:25.478247 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.478221    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"]
Sep 17 18:43:25.482722 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:25.482700    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e4246-predictor-5b57964877-5m2qw"]
Sep 17 18:43:26.159705 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:26.159675    2561 generic.go:356] "Generic (PLEG): container finished" podID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerID="81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc" exitCode=0
Sep 17 18:43:26.160094 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:26.159750    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" event={"ID":"09e96f3f-6976-4259-8bdc-3b69a4a90afc","Type":"ContainerDied","Data":"81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc"}
Sep 17 18:43:27.165850 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:27.165817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" event={"ID":"09e96f3f-6976-4259-8bdc-3b69a4a90afc","Type":"ContainerStarted","Data":"d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c"}
Sep 17 18:43:27.166215 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:27.165877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" event={"ID":"09e96f3f-6976-4259-8bdc-3b69a4a90afc","Type":"ContainerStarted","Data":"b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867"}
Sep 17 18:43:27.166215 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:27.166110    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:27.188169 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:27.188126    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podStartSLOduration=2.616837339 podStartE2EDuration="6.188112808s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.571275469s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:27.186904351 +0000 UTC m=+834.466561380" watchObservedRunningTime="2026-09-17 18:43:27.188112808 +0000 UTC m=+834.467769837"
Sep 17 18:43:27.216742 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:27.216713    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="685172c6-c9b8-40d9-a705-2c613703399c" path="/var/lib/kubelet/pods/685172c6-c9b8-40d9-a705-2c613703399c/volumes"
Sep 17 18:43:28.169638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:28.169606    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:28.170817 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:28.170786    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 18:43:29.177750 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:29.177698    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 18:43:30.115647 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:30.115605    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerName="model-chainer-raw-e4246" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:43:34.093169 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:34.093125    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 17 18:43:34.179041 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:34.179017    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:43:34.179547 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:34.179516    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 18:43:35.114896 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:35.114865    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerName="model-chainer-raw-e4246" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:43:35.115212 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:35.114971    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:40.114840 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:40.114802    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerName="model-chainer-raw-e4246" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:43:44.092422 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:44.092384    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 17 18:43:44.180223 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:44.180186    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 18:43:45.115063 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:45.115026    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerName="model-chainer-raw-e4246" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:43:50.115467 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:50.115417    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerName="model-chainer-raw-e4246" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:43:51.031205 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.031181    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:51.073058 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.073029    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc6f5827-4d2b-410b-96fe-5b912c7c6177-openshift-service-ca-bundle\") pod \"fc6f5827-4d2b-410b-96fe-5b912c7c6177\" (UID: \"fc6f5827-4d2b-410b-96fe-5b912c7c6177\") "
Sep 17 18:43:51.073058 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.073064    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f5827-4d2b-410b-96fe-5b912c7c6177-proxy-tls\") pod \"fc6f5827-4d2b-410b-96fe-5b912c7c6177\" (UID: \"fc6f5827-4d2b-410b-96fe-5b912c7c6177\") "
Sep 17 18:43:51.073389 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.073367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc6f5827-4d2b-410b-96fe-5b912c7c6177-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fc6f5827-4d2b-410b-96fe-5b912c7c6177" (UID: "fc6f5827-4d2b-410b-96fe-5b912c7c6177"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:51.075152 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.075125    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc6f5827-4d2b-410b-96fe-5b912c7c6177-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc6f5827-4d2b-410b-96fe-5b912c7c6177" (UID: "fc6f5827-4d2b-410b-96fe-5b912c7c6177"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:51.174303 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.174223    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc6f5827-4d2b-410b-96fe-5b912c7c6177-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:51.174303 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.174253    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f5827-4d2b-410b-96fe-5b912c7c6177-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:43:51.253568 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.253534    2561 generic.go:356] "Generic (PLEG): container finished" podID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerID="f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab" exitCode=0
Sep 17 18:43:51.253715 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.253610    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"
Sep 17 18:43:51.253715 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.253616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" event={"ID":"fc6f5827-4d2b-410b-96fe-5b912c7c6177","Type":"ContainerDied","Data":"f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab"}
Sep 17 18:43:51.253838 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.253737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj" event={"ID":"fc6f5827-4d2b-410b-96fe-5b912c7c6177","Type":"ContainerDied","Data":"75762f168f8580f71ef80140ab2011d23ce17aab1160c72883e4139e578aaf13"}
Sep 17 18:43:51.253838 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.253758    2561 scope.go:117] "RemoveContainer" containerID="f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab"
Sep 17 18:43:51.261413 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.261398    2561 scope.go:117] "RemoveContainer" containerID="f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab"
Sep 17 18:43:51.261687 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:43:51.261669    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab\": container with ID starting with f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab not found: ID does not exist" containerID="f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab"
Sep 17 18:43:51.261751 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.261694    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab"} err="failed to get container status \"f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab\": rpc error: code = NotFound desc = could not find container \"f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab\": container with ID starting with f5a8363f104cdd8fffd6b51d788d3500326f47e81837c58c615dc27670920cab not found: ID does not exist"
Sep 17 18:43:51.269763 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.269741    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"]
Sep 17 18:43:51.273020 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:51.273001    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e4246-59f9c894c4-ggxrj"]
Sep 17 18:43:53.218188 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:53.218152    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" path="/var/lib/kubelet/pods/fc6f5827-4d2b-410b-96fe-5b912c7c6177/volumes"
Sep 17 18:43:54.092277 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:54.092241    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 17 18:43:54.180413 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:43:54.180372    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 18:44:04.093021 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:04.092981    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 17 18:44:04.180148 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:04.180113    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 18:44:14.093199 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:14.093156    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 17 18:44:14.179960 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:14.179917    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 18:44:15.214490 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:15.214454    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 17 18:44:15.778161 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:15.778124    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:24.179662 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:24.179629    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 18:44:24.278953 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:24.278918    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"]
Sep 17 18:44:24.279158 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:24.279134    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container" containerID="cri-o://0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8" gracePeriod=30
Sep 17 18:44:24.279249 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:24.279211    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kube-rbac-proxy" containerID="cri-o://7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8" gracePeriod=30
Sep 17 18:44:24.279993 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:24.279611    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:44:25.382894 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:25.382856    2561 generic.go:356] "Generic (PLEG): container finished" podID="e61bfd0f-285b-420b-8dac-47759689524c" containerID="7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8" exitCode=2
Sep 17 18:44:25.383257 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:25.382927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" event={"ID":"e61bfd0f-285b-420b-8dac-47759689524c","Type":"ContainerDied","Data":"7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8"}
Sep 17 18:44:28.719341 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.719306    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:44:28.846547 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.846461    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98lwc\" (UniqueName: \"kubernetes.io/projected/e61bfd0f-285b-420b-8dac-47759689524c-kube-api-access-98lwc\") pod \"e61bfd0f-285b-420b-8dac-47759689524c\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") "
Sep 17 18:44:28.846704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.846577    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61bfd0f-285b-420b-8dac-47759689524c-proxy-tls\") pod \"e61bfd0f-285b-420b-8dac-47759689524c\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") "
Sep 17 18:44:28.846704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.846609    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e61bfd0f-285b-420b-8dac-47759689524c-kserve-provision-location\") pod \"e61bfd0f-285b-420b-8dac-47759689524c\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") "
Sep 17 18:44:28.846704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.846649    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-kube-rbac-proxy-sar-config\") pod \"e61bfd0f-285b-420b-8dac-47759689524c\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") "
Sep 17 18:44:28.846704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.846674    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-cabundle-cert\") pod \"e61bfd0f-285b-420b-8dac-47759689524c\" (UID: \"e61bfd0f-285b-420b-8dac-47759689524c\") "
Sep 17 18:44:28.846952 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.846927    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61bfd0f-285b-420b-8dac-47759689524c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e61bfd0f-285b-420b-8dac-47759689524c" (UID: "e61bfd0f-285b-420b-8dac-47759689524c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:28.847035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.846982    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e61bfd0f-285b-420b-8dac-47759689524c" (UID: "e61bfd0f-285b-420b-8dac-47759689524c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:44:28.847126 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.847098    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e61bfd0f-285b-420b-8dac-47759689524c" (UID: "e61bfd0f-285b-420b-8dac-47759689524c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:44:28.848646 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.848625    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61bfd0f-285b-420b-8dac-47759689524c-kube-api-access-98lwc" (OuterVolumeSpecName: "kube-api-access-98lwc") pod "e61bfd0f-285b-420b-8dac-47759689524c" (UID: "e61bfd0f-285b-420b-8dac-47759689524c"). InnerVolumeSpecName "kube-api-access-98lwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:28.848736 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.848642    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61bfd0f-285b-420b-8dac-47759689524c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e61bfd0f-285b-420b-8dac-47759689524c" (UID: "e61bfd0f-285b-420b-8dac-47759689524c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:28.947206 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.947170    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61bfd0f-285b-420b-8dac-47759689524c-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.947206 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.947201    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e61bfd0f-285b-420b-8dac-47759689524c-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.947206 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.947210    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.947394 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.947220    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e61bfd0f-285b-420b-8dac-47759689524c-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:44:28.947394 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:28.947230    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98lwc\" (UniqueName: \"kubernetes.io/projected/e61bfd0f-285b-420b-8dac-47759689524c-kube-api-access-98lwc\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:44:29.407459 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.407424    2561 generic.go:356] "Generic (PLEG): container finished" podID="e61bfd0f-285b-420b-8dac-47759689524c" containerID="0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8" exitCode=0
Sep 17 18:44:29.407631 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.407498    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" event={"ID":"e61bfd0f-285b-420b-8dac-47759689524c","Type":"ContainerDied","Data":"0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8"}
Sep 17 18:44:29.407631 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.407518    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"
Sep 17 18:44:29.407631 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.407572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb" event={"ID":"e61bfd0f-285b-420b-8dac-47759689524c","Type":"ContainerDied","Data":"230bf6eb3916e4780a4ce82b0709bf4d3f99d86d32a7ace8445f994cb6eaf44d"}
Sep 17 18:44:29.407631 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.407599    2561 scope.go:117] "RemoveContainer" containerID="7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8"
Sep 17 18:44:29.415589 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.415570    2561 scope.go:117] "RemoveContainer" containerID="0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8"
Sep 17 18:44:29.422547 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.422528    2561 scope.go:117] "RemoveContainer" containerID="6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1"
Sep 17 18:44:29.424871 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.424853    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"]
Sep 17 18:44:29.429564 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.429544    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-clvgb"]
Sep 17 18:44:29.429976 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.429961    2561 scope.go:117] "RemoveContainer" containerID="7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8"
Sep 17 18:44:29.430264 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:44:29.430245    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8\": container with ID starting with 7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8 not found: ID does not exist" containerID="7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8"
Sep 17 18:44:29.430329 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.430275    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8"} err="failed to get container status \"7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8\": rpc error: code = NotFound desc = could not find container \"7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8\": container with ID starting with 7d00cb52004399236d721c976bab970e3f80f0b20317fada13bc1a1b993ce6b8 not found: ID does not exist"
Sep 17 18:44:29.430329 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.430299    2561 scope.go:117] "RemoveContainer" containerID="0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8"
Sep 17 18:44:29.430594 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:44:29.430577    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8\": container with ID starting with 0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8 not found: ID does not exist" containerID="0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8"
Sep 17 18:44:29.430640 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.430620    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8"} err="failed to get container status \"0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8\": rpc error: code = NotFound desc = could not find container \"0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8\": container with ID starting with 0629d2826a46a1b69ef3663c99266fb23eaf03f11231d33e216227b2fe3352a8 not found: ID does not exist"
Sep 17 18:44:29.430640 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.430637    2561 scope.go:117] "RemoveContainer" containerID="6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1"
Sep 17 18:44:29.430865 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:44:29.430849    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1\": container with ID starting with 6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1 not found: ID does not exist" containerID="6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1"
Sep 17 18:44:29.430907 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:29.430869    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1"} err="failed to get container status \"6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1\": rpc error: code = NotFound desc = could not find container \"6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1\": container with ID starting with 6fb7f30280660450c1ac16df909416850b522ff2fe80879e1cfc6cd34cc775d1 not found: ID does not exist"
Sep 17 18:44:30.954417 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:30.954384    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:31.058295 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:31.058259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:31.217779 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:31.217708    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e61bfd0f-285b-420b-8dac-47759689524c" path="/var/lib/kubelet/pods/e61bfd0f-285b-420b-8dac-47759689524c/volumes"
Sep 17 18:44:34.180422 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:34.180393    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:44:35.084412 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:35.084384    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:44:35.084610 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:35.084594    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:44:41.187423 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.187389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:41.218248 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218220    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"]
Sep 17 18:44:41.218534 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218518    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container"
Sep 17 18:44:41.218621 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218537    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container"
Sep 17 18:44:41.218621 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218551    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kube-rbac-proxy"
Sep 17 18:44:41.218621 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218558    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kube-rbac-proxy"
Sep 17 18:44:41.218621 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218570    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="storage-initializer"
Sep 17 18:44:41.218621 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218578    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="storage-initializer"
Sep 17 18:44:41.218621 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218589    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerName="model-chainer-raw-e4246"
Sep 17 18:44:41.218621 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218597    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerName="model-chainer-raw-e4246"
Sep 17 18:44:41.218621 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218607    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="storage-initializer"
Sep 17 18:44:41.218621 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218617    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="storage-initializer"
Sep 17 18:44:41.219035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218640    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container"
Sep 17 18:44:41.219035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218649    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container"
Sep 17 18:44:41.219035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218670    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kube-rbac-proxy"
Sep 17 18:44:41.219035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218679    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kube-rbac-proxy"
Sep 17 18:44:41.219035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218740    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kserve-container"
Sep 17 18:44:41.219035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218751    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kserve-container"
Sep 17 18:44:41.219035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218762    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="685172c6-c9b8-40d9-a705-2c613703399c" containerName="kube-rbac-proxy"
Sep 17 18:44:41.219035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218771    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc6f5827-4d2b-410b-96fe-5b912c7c6177" containerName="model-chainer-raw-e4246"
Sep 17 18:44:41.219035 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.218779    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61bfd0f-285b-420b-8dac-47759689524c" containerName="kube-rbac-proxy"
Sep 17 18:44:41.223394 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.223375    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.226040 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.226019    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a6bc9a-predictor-serving-cert\""
Sep 17 18:44:41.226146 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.226094    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a6bc9a-kube-rbac-proxy-sar-config\""
Sep 17 18:44:41.233647 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.233624    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"]
Sep 17 18:44:41.337584 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.337554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-cabundle-cert\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.337704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.337591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.337704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.337616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kserve-provision-location\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.337704 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.337678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7289\" (UniqueName: \"kubernetes.io/projected/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-api-access-n7289\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.337810 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.337708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-proxy-tls\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.438474 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.438398    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.438474 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.438455    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kserve-provision-location\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.438656 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.438520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7289\" (UniqueName: \"kubernetes.io/projected/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-api-access-n7289\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.438656 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.438551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-proxy-tls\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.438760 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.438653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-cabundle-cert\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.438899 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.438871    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kserve-provision-location\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.439182 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.439161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.439266 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.439196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-cabundle-cert\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.440983 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.440961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-proxy-tls\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.447301 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.447279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7289\" (UniqueName: \"kubernetes.io/projected/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-api-access-n7289\") pod \"isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") " pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.532696 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.532667    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:41.805259 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:41.805230    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"]
Sep 17 18:44:41.807720 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:44:41.807694    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b23b55_4aa6_4aa8_9d2d_d31f27721d93.slice/crio-3e7a69d559544b7895c67739d8240321408d34f847a21b20de0eb589c8afb786 WatchSource:0}: Error finding container 3e7a69d559544b7895c67739d8240321408d34f847a21b20de0eb589c8afb786: Status 404 returned error can't find the container with id 3e7a69d559544b7895c67739d8240321408d34f847a21b20de0eb589c8afb786
Sep 17 18:44:42.395115 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:42.395063    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:42.456325 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:42.456297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" event={"ID":"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93","Type":"ContainerStarted","Data":"acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b"}
Sep 17 18:44:42.456465 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:42.456355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" event={"ID":"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93","Type":"ContainerStarted","Data":"3e7a69d559544b7895c67739d8240321408d34f847a21b20de0eb589c8afb786"}
Sep 17 18:44:46.472334 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:46.472298    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerID="acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b" exitCode=0
Sep 17 18:44:46.472719 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:46.472369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" event={"ID":"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93","Type":"ContainerDied","Data":"acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b"}
Sep 17 18:44:47.478095 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:47.478061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" event={"ID":"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93","Type":"ContainerStarted","Data":"e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a"}
Sep 17 18:44:47.478496 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:47.478128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" event={"ID":"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93","Type":"ContainerStarted","Data":"61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1"}
Sep 17 18:44:47.478496 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:47.478342    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:47.514021 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:47.513972    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podStartSLOduration=2.7819208399999997 podStartE2EDuration="6.513956901s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.732036061s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:47.513164343 +0000 UTC m=+914.792821372" watchObservedRunningTime="2026-09-17 18:44:47.513956901 +0000 UTC m=+914.793613929"
Sep 17 18:44:48.481845 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:48.481812    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:48.483156 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:48.483126    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 18:44:49.485318 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:49.485278    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 18:44:54.489778 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:54.489746    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:44:54.490310 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:44:54.490278    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 18:45:01.159906 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.159873    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"]
Sep 17 18:45:01.163149 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.163134    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:01.166111 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.166075    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-7a45d-kube-rbac-proxy-sar-config\""
Sep 17 18:45:01.166231 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.166194    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-7a45d-serving-cert\""
Sep 17 18:45:01.172816 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.172792    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"]
Sep 17 18:45:01.173812 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.173787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:01.274963 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.274930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m\" (UID: \"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:01.274963 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.274964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-proxy-tls\") pod \"model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m\" (UID: \"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:01.354945 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.354913    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:01.375996 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.375970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m\" (UID: \"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:01.376107 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.376004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-proxy-tls\") pod \"model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m\" (UID: \"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:01.376695 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.376673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m\" (UID: \"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:01.378306 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.378290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-proxy-tls\") pod \"model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m\" (UID: \"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:01.473233 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.473145    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:01.740919 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:01.740897    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"]
Sep 17 18:45:01.743140 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:45:01.743113    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb48d344_f89d_4e1c_87b5_1e5542f7ffe7.slice/crio-1ca85639154cf877f8bcd401e189ce00b805f945ad8f9493413691412a0888e9 WatchSource:0}: Error finding container 1ca85639154cf877f8bcd401e189ce00b805f945ad8f9493413691412a0888e9: Status 404 returned error can't find the container with id 1ca85639154cf877f8bcd401e189ce00b805f945ad8f9493413691412a0888e9
Sep 17 18:45:02.531935 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:02.531898    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" event={"ID":"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7","Type":"ContainerStarted","Data":"7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6"}
Sep 17 18:45:02.532346 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:02.531979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" event={"ID":"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7","Type":"ContainerStarted","Data":"1ca85639154cf877f8bcd401e189ce00b805f945ad8f9493413691412a0888e9"}
Sep 17 18:45:02.532346 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:02.532040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:02.549706 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:02.549661    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" podStartSLOduration=1.5496467310000002 podStartE2EDuration="1.549646731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:02.548355831 +0000 UTC m=+929.828012861" watchObservedRunningTime="2026-09-17 18:45:02.549646731 +0000 UTC m=+929.829303760"
Sep 17 18:45:04.490891 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:04.490855    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 18:45:08.540834 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:08.540806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:11.241855 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:11.241822    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"]
Sep 17 18:45:11.242289 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:11.242049    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerName="model-chainer-raw-hpa-7a45d" containerID="cri-o://7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6" gracePeriod=30
Sep 17 18:45:11.283393 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:11.283364    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.404143 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:11.404105    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.483730 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:11.483697    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"]
Sep 17 18:45:11.483948 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:11.483928    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container" containerID="cri-o://b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867" gracePeriod=30
Sep 17 18:45:11.484019 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:11.483987    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kube-rbac-proxy" containerID="cri-o://d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c" gracePeriod=30
Sep 17 18:45:11.604034 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:11.604008    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.697670 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:11.697620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.793695 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:11.793613    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:12.569329 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:12.569300    2561 generic.go:356] "Generic (PLEG): container finished" podID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerID="d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c" exitCode=2
Sep 17 18:45:12.569776 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:12.569373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" event={"ID":"09e96f3f-6976-4259-8bdc-3b69a4a90afc","Type":"ContainerDied","Data":"d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c"}
Sep 17 18:45:13.539747 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:13.539711    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerName="model-chainer-raw-hpa-7a45d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:45:14.175585 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:14.175543    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Sep 17 18:45:14.179806 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:14.179782    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 18:45:14.490569 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:14.490467    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 18:45:15.218379 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.218355    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:45:15.280277 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.280251    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-rbac-proxy-sar-config\") pod \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") "
Sep 17 18:45:15.280407 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.280288    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x68zn\" (UniqueName: \"kubernetes.io/projected/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-api-access-x68zn\") pod \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") "
Sep 17 18:45:15.280407 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.280313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e96f3f-6976-4259-8bdc-3b69a4a90afc-proxy-tls\") pod \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") "
Sep 17 18:45:15.280407 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.280346    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-cabundle-cert\") pod \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") "
Sep 17 18:45:15.280407 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.280378    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kserve-provision-location\") pod \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\" (UID: \"09e96f3f-6976-4259-8bdc-3b69a4a90afc\") "
Sep 17 18:45:15.281097 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.280748    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09e96f3f-6976-4259-8bdc-3b69a4a90afc" (UID: "09e96f3f-6976-4259-8bdc-3b69a4a90afc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:15.281097 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.280758    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09e96f3f-6976-4259-8bdc-3b69a4a90afc" (UID: "09e96f3f-6976-4259-8bdc-3b69a4a90afc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:15.281233 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.281201    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09e96f3f-6976-4259-8bdc-3b69a4a90afc" (UID: "09e96f3f-6976-4259-8bdc-3b69a4a90afc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:15.282453 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.282431    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-api-access-x68zn" (OuterVolumeSpecName: "kube-api-access-x68zn") pod "09e96f3f-6976-4259-8bdc-3b69a4a90afc" (UID: "09e96f3f-6976-4259-8bdc-3b69a4a90afc"). InnerVolumeSpecName "kube-api-access-x68zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:15.282561 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.282442    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e96f3f-6976-4259-8bdc-3b69a4a90afc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09e96f3f-6976-4259-8bdc-3b69a4a90afc" (UID: "09e96f3f-6976-4259-8bdc-3b69a4a90afc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:15.381057 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.381030    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:45:15.381057 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.381054    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x68zn\" (UniqueName: \"kubernetes.io/projected/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kube-api-access-x68zn\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:45:15.381240 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.381067    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e96f3f-6976-4259-8bdc-3b69a4a90afc-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:45:15.381240 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.381080    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e96f3f-6976-4259-8bdc-3b69a4a90afc-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:45:15.381240 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.381090    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e96f3f-6976-4259-8bdc-3b69a4a90afc-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:45:15.582049 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.582015    2561 generic.go:356] "Generic (PLEG): container finished" podID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerID="b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867" exitCode=0
Sep 17 18:45:15.582219 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.582079    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"
Sep 17 18:45:15.582219 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.582090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" event={"ID":"09e96f3f-6976-4259-8bdc-3b69a4a90afc","Type":"ContainerDied","Data":"b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867"}
Sep 17 18:45:15.582219 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.582150    2561 scope.go:117] "RemoveContainer" containerID="d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c"
Sep 17 18:45:15.587233 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.582562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t" event={"ID":"09e96f3f-6976-4259-8bdc-3b69a4a90afc","Type":"ContainerDied","Data":"ab21bfa92b8577878ed58c0748efe9a3c9cbd9fef8ab9c80661af17a9111f64f"}
Sep 17 18:45:15.595696 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.595679    2561 scope.go:117] "RemoveContainer" containerID="b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867"
Sep 17 18:45:15.602841 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.602824    2561 scope.go:117] "RemoveContainer" containerID="81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc"
Sep 17 18:45:15.608079 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.608058    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"]
Sep 17 18:45:15.609751 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.609737    2561 scope.go:117] "RemoveContainer" containerID="d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c"
Sep 17 18:45:15.610001 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:45:15.609982    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c\": container with ID starting with d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c not found: ID does not exist" containerID="d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c"
Sep 17 18:45:15.610123 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.610007    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c"} err="failed to get container status \"d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c\": rpc error: code = NotFound desc = could not find container \"d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c\": container with ID starting with d8ab9ede1f2a7c414d7be576a7408cc63980d13dd6ca5f2fb2c8bdb7c426de6c not found: ID does not exist"
Sep 17 18:45:15.610123 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.610024    2561 scope.go:117] "RemoveContainer" containerID="b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867"
Sep 17 18:45:15.610378 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:45:15.610283    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867\": container with ID starting with b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867 not found: ID does not exist" containerID="b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867"
Sep 17 18:45:15.610378 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.610317    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867"} err="failed to get container status \"b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867\": rpc error: code = NotFound desc = could not find container \"b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867\": container with ID starting with b1d98d9bbca49ff5da44cfb9df484ed8bf0d05491e4b057c92dd5731fc699867 not found: ID does not exist"
Sep 17 18:45:15.610378 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.610351    2561 scope.go:117] "RemoveContainer" containerID="81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc"
Sep 17 18:45:15.610825 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:45:15.610798    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc\": container with ID starting with 81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc not found: ID does not exist" containerID="81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc"
Sep 17 18:45:15.610907 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.610830    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc"} err="failed to get container status \"81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc\": rpc error: code = NotFound desc = could not find container \"81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc\": container with ID starting with 81b57a7b473104bceb741ac06b91c5207a27eeb432001fe12b0c7c7a3e4f67fc not found: ID does not exist"
Sep 17 18:45:15.611774 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:15.611756    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7a45d-predictor-695b9c6989-qck2t"]
Sep 17 18:45:17.217779 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:17.217742    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" path="/var/lib/kubelet/pods/09e96f3f-6976-4259-8bdc-3b69a4a90afc/volumes"
Sep 17 18:45:18.539467 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:18.539425    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerName="model-chainer-raw-hpa-7a45d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:45:23.539014 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:23.538973    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerName="model-chainer-raw-hpa-7a45d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:45:23.539392 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:23.539093    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:24.491137 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:24.491098    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 18:45:28.539763 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:28.539721    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerName="model-chainer-raw-hpa-7a45d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:45:31.453091 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:31.453030    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:31.796387 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:31.796301    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:33.539288 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:33.539248    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerName="model-chainer-raw-hpa-7a45d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:45:34.490828 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:34.490787    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 18:45:38.539194 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:38.539158    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerName="model-chainer-raw-hpa-7a45d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:45:41.384283 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.384259    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:41.458892 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.458860    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-proxy-tls\") pod \"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7\" (UID: \"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7\") "
Sep 17 18:45:41.459055 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.458923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-openshift-service-ca-bundle\") pod \"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7\" (UID: \"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7\") "
Sep 17 18:45:41.459256 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.459231    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" (UID: "bb48d344-f89d-4e1c-87b5-1e5542f7ffe7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:41.460973 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.460948    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" (UID: "bb48d344-f89d-4e1c-87b5-1e5542f7ffe7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:41.560207 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.560173    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:45:41.560207 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.560204    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:45:41.673400 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.673362    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerID="7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6" exitCode=0
Sep 17 18:45:41.673614 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.673407    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"
Sep 17 18:45:41.673614 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.673422    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" event={"ID":"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7","Type":"ContainerDied","Data":"7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6"}
Sep 17 18:45:41.673614 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.673498    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m" event={"ID":"bb48d344-f89d-4e1c-87b5-1e5542f7ffe7","Type":"ContainerDied","Data":"1ca85639154cf877f8bcd401e189ce00b805f945ad8f9493413691412a0888e9"}
Sep 17 18:45:41.673614 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.673529    2561 scope.go:117] "RemoveContainer" containerID="7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6"
Sep 17 18:45:41.682053 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.682031    2561 scope.go:117] "RemoveContainer" containerID="7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6"
Sep 17 18:45:41.682336 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:45:41.682317    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6\": container with ID starting with 7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6 not found: ID does not exist" containerID="7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6"
Sep 17 18:45:41.682390 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.682348    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6"} err="failed to get container status \"7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6\": rpc error: code = NotFound desc = could not find container \"7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6\": container with ID starting with 7f718cb4a1c39a75efa7ff771a261151f8125fcaafc4c312a700213811371fb6 not found: ID does not exist"
Sep 17 18:45:41.694569 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.694538    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"]
Sep 17 18:45:41.699170 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:41.699148    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-7a45d-d8d9bf78b-jrk6m"]
Sep 17 18:45:43.217055 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:43.217011    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" path="/var/lib/kubelet/pods/bb48d344-f89d-4e1c-87b5-1e5542f7ffe7/volumes"
Sep 17 18:45:44.490960 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:44.490922    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 18:45:54.490270 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:45:54.490190    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 18:46:03.922782 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:03.922738    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:04.490636 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:04.490599    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:46:26.495774 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.495735    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:26.559452 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559422    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"]
Sep 17 18:46:26.559715 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559701    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="storage-initializer"
Sep 17 18:46:26.559757 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559717    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="storage-initializer"
Sep 17 18:46:26.559757 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559727    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container"
Sep 17 18:46:26.559757 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559734    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container"
Sep 17 18:46:26.559757 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559743    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kube-rbac-proxy"
Sep 17 18:46:26.559757 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559751    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kube-rbac-proxy"
Sep 17 18:46:26.559904 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559779    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerName="model-chainer-raw-hpa-7a45d"
Sep 17 18:46:26.559904 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559785    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerName="model-chainer-raw-hpa-7a45d"
Sep 17 18:46:26.559904 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559843    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kube-rbac-proxy"
Sep 17 18:46:26.559904 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559852    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb48d344-f89d-4e1c-87b5-1e5542f7ffe7" containerName="model-chainer-raw-hpa-7a45d"
Sep 17 18:46:26.559904 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.559858    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e96f3f-6976-4259-8bdc-3b69a4a90afc" containerName="kserve-container"
Sep 17 18:46:26.562631 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.562616    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.569358 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.569341    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-a6bc9a\""
Sep 17 18:46:26.569453 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.569346    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a6bc9a-predictor-serving-cert\""
Sep 17 18:46:26.570285 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.570267    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a6bc9a-kube-rbac-proxy-sar-config\""
Sep 17 18:46:26.570285 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.570277    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-a6bc9a-dockercfg-sjt8t\""
Sep 17 18:46:26.606894 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.606869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"]
Sep 17 18:46:26.698295 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.698270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e45c836-1505-4e15-b26e-714da80dd5a6-proxy-tls\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.698423 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.698307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znhqh\" (UniqueName: \"kubernetes.io/projected/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-api-access-znhqh\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.698423 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.698339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.698519 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.698417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-cabundle-cert\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.698519 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.698461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e45c836-1505-4e15-b26e-714da80dd5a6-kserve-provision-location\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.782210 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.782145    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:26.799211 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.799188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.799303 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.799221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-cabundle-cert\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.799303 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.799257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e45c836-1505-4e15-b26e-714da80dd5a6-kserve-provision-location\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.799303 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.799293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e45c836-1505-4e15-b26e-714da80dd5a6-proxy-tls\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.799412 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.799316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znhqh\" (UniqueName: \"kubernetes.io/projected/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-api-access-znhqh\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.799784 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.799762    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e45c836-1505-4e15-b26e-714da80dd5a6-kserve-provision-location\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.800025 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.800005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.800097 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.800006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-cabundle-cert\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.801740 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.801722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e45c836-1505-4e15-b26e-714da80dd5a6-proxy-tls\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.808457 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.808430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znhqh\" (UniqueName: \"kubernetes.io/projected/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-api-access-znhqh\") pod \"isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") " pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:26.872791 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:26.872763    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:27.149288 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:27.149186    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"]
Sep 17 18:46:27.152052 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:46:27.152014    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e45c836_1505_4e15_b26e_714da80dd5a6.slice/crio-d8cae782957d22f8e88aeed6750d783dcec685e4e4013054927249969e05ef5b WatchSource:0}: Error finding container d8cae782957d22f8e88aeed6750d783dcec685e4e4013054927249969e05ef5b: Status 404 returned error can't find the container with id d8cae782957d22f8e88aeed6750d783dcec685e4e4013054927249969e05ef5b
Sep 17 18:46:27.835539 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:27.835499    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5" event={"ID":"5e45c836-1505-4e15-b26e-714da80dd5a6","Type":"ContainerStarted","Data":"c2a59a520ddd9d6e2097263f45a223e490556ede8663e517f4793f44a3fdf709"}
Sep 17 18:46:27.835899 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:27.835572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5" event={"ID":"5e45c836-1505-4e15-b26e-714da80dd5a6","Type":"ContainerStarted","Data":"d8cae782957d22f8e88aeed6750d783dcec685e4e4013054927249969e05ef5b"}
Sep 17 18:46:31.851290 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:31.851258    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5_5e45c836-1505-4e15-b26e-714da80dd5a6/storage-initializer/0.log"
Sep 17 18:46:31.851665 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:31.851300    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e45c836-1505-4e15-b26e-714da80dd5a6" containerID="c2a59a520ddd9d6e2097263f45a223e490556ede8663e517f4793f44a3fdf709" exitCode=1
Sep 17 18:46:31.851665 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:31.851364    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5" event={"ID":"5e45c836-1505-4e15-b26e-714da80dd5a6","Type":"ContainerDied","Data":"c2a59a520ddd9d6e2097263f45a223e490556ede8663e517f4793f44a3fdf709"}
Sep 17 18:46:32.857241 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:32.857214    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5_5e45c836-1505-4e15-b26e-714da80dd5a6/storage-initializer/0.log"
Sep 17 18:46:32.857614 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:32.857275    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5" event={"ID":"5e45c836-1505-4e15-b26e-714da80dd5a6","Type":"ContainerStarted","Data":"3e4ee3fd3e0d5c5194fa650354a100a201de0435fc4141641376bf8754e75cec"}
Sep 17 18:46:35.869660 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:35.869628    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5_5e45c836-1505-4e15-b26e-714da80dd5a6/storage-initializer/1.log"
Sep 17 18:46:35.870060 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:35.870044    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5_5e45c836-1505-4e15-b26e-714da80dd5a6/storage-initializer/0.log"
Sep 17 18:46:35.870103 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:35.870078    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e45c836-1505-4e15-b26e-714da80dd5a6" containerID="3e4ee3fd3e0d5c5194fa650354a100a201de0435fc4141641376bf8754e75cec" exitCode=1
Sep 17 18:46:35.870154 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:35.870141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5" event={"ID":"5e45c836-1505-4e15-b26e-714da80dd5a6","Type":"ContainerDied","Data":"3e4ee3fd3e0d5c5194fa650354a100a201de0435fc4141641376bf8754e75cec"}
Sep 17 18:46:35.870209 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:35.870199    2561 scope.go:117] "RemoveContainer" containerID="c2a59a520ddd9d6e2097263f45a223e490556ede8663e517f4793f44a3fdf709"
Sep 17 18:46:35.870594 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:35.870572    2561 scope.go:117] "RemoveContainer" containerID="c2a59a520ddd9d6e2097263f45a223e490556ede8663e517f4793f44a3fdf709"
Sep 17 18:46:35.881042 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:46:35.881018    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5_kserve-ci-e2e-test(5e45c836-1505-4e15-b26e-714da80dd5a6)\"" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5" podUID="5e45c836-1505-4e15-b26e-714da80dd5a6"
Sep 17 18:46:35.881527 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:46:35.881446    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5_kserve-ci-e2e-test_5e45c836-1505-4e15-b26e-714da80dd5a6_0 in pod sandbox d8cae782957d22f8e88aeed6750d783dcec685e4e4013054927249969e05ef5b from index: no such id: 'c2a59a520ddd9d6e2097263f45a223e490556ede8663e517f4793f44a3fdf709'" containerID="c2a59a520ddd9d6e2097263f45a223e490556ede8663e517f4793f44a3fdf709"
Sep 17 18:46:35.881527 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:35.881501    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a59a520ddd9d6e2097263f45a223e490556ede8663e517f4793f44a3fdf709"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5_kserve-ci-e2e-test_5e45c836-1505-4e15-b26e-714da80dd5a6_0 in pod sandbox d8cae782957d22f8e88aeed6750d783dcec685e4e4013054927249969e05ef5b from index: no such id: 'c2a59a520ddd9d6e2097263f45a223e490556ede8663e517f4793f44a3fdf709'"
Sep 17 18:46:36.875592 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:36.875565    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5_5e45c836-1505-4e15-b26e-714da80dd5a6/storage-initializer/1.log"
Sep 17 18:46:42.543588 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.543556    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"]
Sep 17 18:46:42.595815 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.595706    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:42.598291 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.598266    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"]
Sep 17 18:46:42.598613 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.598568    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" containerID="cri-o://61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1" gracePeriod=30
Sep 17 18:46:42.598681 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.598607    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kube-rbac-proxy" containerID="cri-o://e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a" gracePeriod=30
Sep 17 18:46:42.717048 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.717026    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5_5e45c836-1505-4e15-b26e-714da80dd5a6/storage-initializer/1.log"
Sep 17 18:46:42.719281 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.719263    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:42.721890 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.721866    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:42.814673 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.814641    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e45c836-1505-4e15-b26e-714da80dd5a6-kserve-provision-location\") pod \"5e45c836-1505-4e15-b26e-714da80dd5a6\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") "
Sep 17 18:46:42.814870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.814694    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-rbac-proxy-sar-config\") pod \"5e45c836-1505-4e15-b26e-714da80dd5a6\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") "
Sep 17 18:46:42.814870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.814722    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znhqh\" (UniqueName: \"kubernetes.io/projected/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-api-access-znhqh\") pod \"5e45c836-1505-4e15-b26e-714da80dd5a6\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") "
Sep 17 18:46:42.814870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.814739    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-cabundle-cert\") pod \"5e45c836-1505-4e15-b26e-714da80dd5a6\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") "
Sep 17 18:46:42.814870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.814777    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e45c836-1505-4e15-b26e-714da80dd5a6-proxy-tls\") pod \"5e45c836-1505-4e15-b26e-714da80dd5a6\" (UID: \"5e45c836-1505-4e15-b26e-714da80dd5a6\") "
Sep 17 18:46:42.815089 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.814922    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e45c836-1505-4e15-b26e-714da80dd5a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e45c836-1505-4e15-b26e-714da80dd5a6" (UID: "5e45c836-1505-4e15-b26e-714da80dd5a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:42.815145 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.815085    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e45c836-1505-4e15-b26e-714da80dd5a6" (UID: "5e45c836-1505-4e15-b26e-714da80dd5a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:42.815145 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.815120    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e45c836-1505-4e15-b26e-714da80dd5a6" (UID: "5e45c836-1505-4e15-b26e-714da80dd5a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:42.816998 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.816977    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-api-access-znhqh" (OuterVolumeSpecName: "kube-api-access-znhqh") pod "5e45c836-1505-4e15-b26e-714da80dd5a6" (UID: "5e45c836-1505-4e15-b26e-714da80dd5a6"). InnerVolumeSpecName "kube-api-access-znhqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:42.816998 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.816986    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e45c836-1505-4e15-b26e-714da80dd5a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e45c836-1505-4e15-b26e-714da80dd5a6" (UID: "5e45c836-1505-4e15-b26e-714da80dd5a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:42.897338 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.897309    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerID="e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a" exitCode=2
Sep 17 18:46:42.897516 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.897389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" event={"ID":"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93","Type":"ContainerDied","Data":"e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a"}
Sep 17 18:46:42.898668 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.898651    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5_5e45c836-1505-4e15-b26e-714da80dd5a6/storage-initializer/1.log"
Sep 17 18:46:42.898738 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.898697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5" event={"ID":"5e45c836-1505-4e15-b26e-714da80dd5a6","Type":"ContainerDied","Data":"d8cae782957d22f8e88aeed6750d783dcec685e4e4013054927249969e05ef5b"}
Sep 17 18:46:42.898787 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.898756    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"
Sep 17 18:46:42.898846 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.898758    2561 scope.go:117] "RemoveContainer" containerID="3e4ee3fd3e0d5c5194fa650354a100a201de0435fc4141641376bf8754e75cec"
Sep 17 18:46:42.916094 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.916069    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e45c836-1505-4e15-b26e-714da80dd5a6-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:42.916094 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.916090    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:42.916210 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.916100    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znhqh\" (UniqueName: \"kubernetes.io/projected/5e45c836-1505-4e15-b26e-714da80dd5a6-kube-api-access-znhqh\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:42.916210 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.916110    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e45c836-1505-4e15-b26e-714da80dd5a6-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:42.916210 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.916118    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e45c836-1505-4e15-b26e-714da80dd5a6-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:42.939605 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.939582    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"]
Sep 17 18:46:42.947798 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:42.947765    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a6bc9a-predictor-5ccf7c5779-vb2p5"]
Sep 17 18:46:43.217127 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:43.217053    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e45c836-1505-4e15-b26e-714da80dd5a6" path="/var/lib/kubelet/pods/5e45c836-1505-4e15-b26e-714da80dd5a6/volumes"
Sep 17 18:46:44.486130 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:44.486090    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 17 18:46:44.490405 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:44.490378    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 18:46:47.336991 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.336965    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:46:47.347209 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.347183    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-cabundle-cert\") pod \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") "
Sep 17 18:46:47.347338 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.347225    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kserve-provision-location\") pod \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") "
Sep 17 18:46:47.347338 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.347253    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7289\" (UniqueName: \"kubernetes.io/projected/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-api-access-n7289\") pod \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") "
Sep 17 18:46:47.347338 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.347282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-proxy-tls\") pod \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") "
Sep 17 18:46:47.347338 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.347329    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-rbac-proxy-sar-config\") pod \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\" (UID: \"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93\") "
Sep 17 18:46:47.347558 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.347535    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" (UID: "b5b23b55-4aa6-4aa8-9d2d-d31f27721d93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:47.347681 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.347661    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" (UID: "b5b23b55-4aa6-4aa8-9d2d-d31f27721d93"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:47.347737 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.347714    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" (UID: "b5b23b55-4aa6-4aa8-9d2d-d31f27721d93"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:47.349292 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.349269    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-api-access-n7289" (OuterVolumeSpecName: "kube-api-access-n7289") pod "b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" (UID: "b5b23b55-4aa6-4aa8-9d2d-d31f27721d93"). InnerVolumeSpecName "kube-api-access-n7289". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:47.349377 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.349291    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" (UID: "b5b23b55-4aa6-4aa8-9d2d-d31f27721d93"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:47.448558 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.448453    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:47.448558 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.448501    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7289\" (UniqueName: \"kubernetes.io/projected/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-api-access-n7289\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:47.448558 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.448511    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:47.448558 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.448522    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:47.448558 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.448532    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:47.855179 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.855139    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:47.904200 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904165    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"]
Sep 17 18:46:47.904411 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904398    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e45c836-1505-4e15-b26e-714da80dd5a6" containerName="storage-initializer"
Sep 17 18:46:47.904449 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904412    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e45c836-1505-4e15-b26e-714da80dd5a6" containerName="storage-initializer"
Sep 17 18:46:47.904449 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904426    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container"
Sep 17 18:46:47.904449 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904432    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container"
Sep 17 18:46:47.904449 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904444    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kube-rbac-proxy"
Sep 17 18:46:47.904449 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904451    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kube-rbac-proxy"
Sep 17 18:46:47.904637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904463    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="storage-initializer"
Sep 17 18:46:47.904637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904469    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="storage-initializer"
Sep 17 18:46:47.904637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904531    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e45c836-1505-4e15-b26e-714da80dd5a6" containerName="storage-initializer"
Sep 17 18:46:47.904637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904544    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e45c836-1505-4e15-b26e-714da80dd5a6" containerName="storage-initializer"
Sep 17 18:46:47.904637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904551    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kserve-container"
Sep 17 18:46:47.904637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904558    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerName="kube-rbac-proxy"
Sep 17 18:46:47.904637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904595    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e45c836-1505-4e15-b26e-714da80dd5a6" containerName="storage-initializer"
Sep 17 18:46:47.904637 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.904601    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e45c836-1505-4e15-b26e-714da80dd5a6" containerName="storage-initializer"
Sep 17 18:46:47.907709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.907694    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:47.910119 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.910101    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-d7124a\""
Sep 17 18:46:47.910230 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.910191    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-d7124a-predictor-serving-cert\""
Sep 17 18:46:47.910426 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.910401    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-d7124a-kube-rbac-proxy-sar-config\""
Sep 17 18:46:47.910753 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.910736    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-d7124a-dockercfg-65lm5\""
Sep 17 18:46:47.919198 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.919174    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" containerID="61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1" exitCode=0
Sep 17 18:46:47.919285 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.919230    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"
Sep 17 18:46:47.919285 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.919238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" event={"ID":"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93","Type":"ContainerDied","Data":"61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1"}
Sep 17 18:46:47.919358 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.919325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n" event={"ID":"b5b23b55-4aa6-4aa8-9d2d-d31f27721d93","Type":"ContainerDied","Data":"3e7a69d559544b7895c67739d8240321408d34f847a21b20de0eb589c8afb786"}
Sep 17 18:46:47.919398 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.919364    2561 scope.go:117] "RemoveContainer" containerID="e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a"
Sep 17 18:46:47.925044 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.925023    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"]
Sep 17 18:46:47.927747 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.927729    2561 scope.go:117] "RemoveContainer" containerID="61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1"
Sep 17 18:46:47.936175 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.936158    2561 scope.go:117] "RemoveContainer" containerID="acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b"
Sep 17 18:46:47.943062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.943046    2561 scope.go:117] "RemoveContainer" containerID="e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a"
Sep 17 18:46:47.943319 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:46:47.943300    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a\": container with ID starting with e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a not found: ID does not exist" containerID="e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a"
Sep 17 18:46:47.943377 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.943327    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a"} err="failed to get container status \"e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a\": rpc error: code = NotFound desc = could not find container \"e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a\": container with ID starting with e28da61225f3bf547a92a81a469ee9d6b4b189c480281f4cf28e2401f91f934a not found: ID does not exist"
Sep 17 18:46:47.943377 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.943348    2561 scope.go:117] "RemoveContainer" containerID="61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1"
Sep 17 18:46:47.943640 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:46:47.943621    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1\": container with ID starting with 61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1 not found: ID does not exist" containerID="61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1"
Sep 17 18:46:47.943703 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.943653    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1"} err="failed to get container status \"61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1\": rpc error: code = NotFound desc = could not find container \"61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1\": container with ID starting with 61f5c9767f4374a716bd2195015f74061f75606ec3e94ed93bb0fcc2d73e7ad1 not found: ID does not exist"
Sep 17 18:46:47.943703 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.943668    2561 scope.go:117] "RemoveContainer" containerID="acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b"
Sep 17 18:46:47.943910 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:46:47.943894    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b\": container with ID starting with acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b not found: ID does not exist" containerID="acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b"
Sep 17 18:46:47.943950 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.943915    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b"} err="failed to get container status \"acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b\": rpc error: code = NotFound desc = could not find container \"acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b\": container with ID starting with acac5582afbeaba34721d9874dd2ed566e89b1e0ea3cf3421a4c4e252002b32b not found: ID does not exist"
Sep 17 18:46:47.951985 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.951964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-proxy-tls\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:47.952077 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.952001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:47.952077 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.952022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-cabundle-cert\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:47.952148 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.952076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5mz\" (UniqueName: \"kubernetes.io/projected/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-api-access-qn5mz\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:47.952148 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.952098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kserve-provision-location\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:47.969939 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.969903    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"]
Sep 17 18:46:47.974949 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:47.974925    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a6bc9a-predictor-65bbd7676d-rhs2n"]
Sep 17 18:46:48.053374 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.053329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kserve-provision-location\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.053570 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.053385    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-proxy-tls\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.053570 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.053424    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.053570 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.053448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-cabundle-cert\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.053570 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.053505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5mz\" (UniqueName: \"kubernetes.io/projected/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-api-access-qn5mz\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.053782 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:46:48.053608    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-serving-cert: secret "isvc-init-fail-d7124a-predictor-serving-cert" not found
Sep 17 18:46:48.053782 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:46:48.053692    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-proxy-tls podName:3fe75e38-dc7e-42f4-9513-e8ad13c90d81 nodeName:}" failed. No retries permitted until 2026-09-17 18:46:48.553667796 +0000 UTC m=+1035.833324803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-proxy-tls") pod "isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl" (UID: "3fe75e38-dc7e-42f4-9513-e8ad13c90d81") : secret "isvc-init-fail-d7124a-predictor-serving-cert" not found
Sep 17 18:46:48.053854 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.053816    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kserve-provision-location\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.054116 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.054097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.054151 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.054101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-cabundle-cert\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.065559 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.065540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5mz\" (UniqueName: \"kubernetes.io/projected/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-api-access-qn5mz\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.136965 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.136870    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:48.557408 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.557375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-proxy-tls\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.559819 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.559788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-proxy-tls\") pod \"isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") " pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:48.818235 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:48.818147    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:49.092760 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:49.092623    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"]
Sep 17 18:46:49.095270 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:46:49.095241    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe75e38_dc7e_42f4_9513_e8ad13c90d81.slice/crio-beea34d6fa52ac8cb5e9139cae5fe88c9870bc74f6baf1bea1085dc124ee5fd1 WatchSource:0}: Error finding container beea34d6fa52ac8cb5e9139cae5fe88c9870bc74f6baf1bea1085dc124ee5fd1: Status 404 returned error can't find the container with id beea34d6fa52ac8cb5e9139cae5fe88c9870bc74f6baf1bea1085dc124ee5fd1
Sep 17 18:46:49.218800 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:49.218768    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b23b55-4aa6-4aa8-9d2d-d31f27721d93" path="/var/lib/kubelet/pods/b5b23b55-4aa6-4aa8-9d2d-d31f27721d93/volumes"
Sep 17 18:46:49.929399 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:49.929359    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl" event={"ID":"3fe75e38-dc7e-42f4-9513-e8ad13c90d81","Type":"ContainerStarted","Data":"5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc"}
Sep 17 18:46:49.929878 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:49.929432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl" event={"ID":"3fe75e38-dc7e-42f4-9513-e8ad13c90d81","Type":"ContainerStarted","Data":"beea34d6fa52ac8cb5e9139cae5fe88c9870bc74f6baf1bea1085dc124ee5fd1"}
Sep 17 18:46:52.941492 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:52.941442    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl_3fe75e38-dc7e-42f4-9513-e8ad13c90d81/storage-initializer/0.log"
Sep 17 18:46:52.941870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:52.941515    2561 generic.go:356] "Generic (PLEG): container finished" podID="3fe75e38-dc7e-42f4-9513-e8ad13c90d81" containerID="5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc" exitCode=1
Sep 17 18:46:52.941870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:52.941588    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl" event={"ID":"3fe75e38-dc7e-42f4-9513-e8ad13c90d81","Type":"ContainerDied","Data":"5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc"}
Sep 17 18:46:53.947057 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:53.947030    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl_3fe75e38-dc7e-42f4-9513-e8ad13c90d81/storage-initializer/0.log"
Sep 17 18:46:53.947463 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:53.947100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl" event={"ID":"3fe75e38-dc7e-42f4-9513-e8ad13c90d81","Type":"ContainerStarted","Data":"15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8"}
Sep 17 18:46:57.754759 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:57.754722    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"]
Sep 17 18:46:57.755259 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:57.754949    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl" podUID="3fe75e38-dc7e-42f4-9513-e8ad13c90d81" containerName="storage-initializer" containerID="cri-o://15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8" gracePeriod=30
Sep 17 18:46:57.780239 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:57.780175    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:58.093694 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.093671    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl_3fe75e38-dc7e-42f4-9513-e8ad13c90d81/storage-initializer/1.log"
Sep 17 18:46:58.094103 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.094087    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl_3fe75e38-dc7e-42f4-9513-e8ad13c90d81/storage-initializer/0.log"
Sep 17 18:46:58.096449 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.096435    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:58.130824 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.130787    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-rbac-proxy-sar-config\") pod \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") "
Sep 17 18:46:58.130949 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.130846    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kserve-provision-location\") pod \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") "
Sep 17 18:46:58.130949 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.130876    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-cabundle-cert\") pod \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") "
Sep 17 18:46:58.130949 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.130912    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-proxy-tls\") pod \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") "
Sep 17 18:46:58.130949 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.130931    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn5mz\" (UniqueName: \"kubernetes.io/projected/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-api-access-qn5mz\") pod \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\" (UID: \"3fe75e38-dc7e-42f4-9513-e8ad13c90d81\") "
Sep 17 18:46:58.131176 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.131151    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fe75e38-dc7e-42f4-9513-e8ad13c90d81" (UID: "3fe75e38-dc7e-42f4-9513-e8ad13c90d81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:58.131221 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.131170    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3fe75e38-dc7e-42f4-9513-e8ad13c90d81" (UID: "3fe75e38-dc7e-42f4-9513-e8ad13c90d81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:58.131266 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.131234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3fe75e38-dc7e-42f4-9513-e8ad13c90d81" (UID: "3fe75e38-dc7e-42f4-9513-e8ad13c90d81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:58.132930 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.132898    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3fe75e38-dc7e-42f4-9513-e8ad13c90d81" (UID: "3fe75e38-dc7e-42f4-9513-e8ad13c90d81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:58.132930 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.132908    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-api-access-qn5mz" (OuterVolumeSpecName: "kube-api-access-qn5mz") pod "3fe75e38-dc7e-42f4-9513-e8ad13c90d81" (UID: "3fe75e38-dc7e-42f4-9513-e8ad13c90d81"). InnerVolumeSpecName "kube-api-access-qn5mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:58.232152 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.232124    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:58.232152 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.232148    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:58.232317 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.232163    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:58.232317 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.232174    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn5mz\" (UniqueName: \"kubernetes.io/projected/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-api-access-qn5mz\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:58.232317 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.232186    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fe75e38-dc7e-42f4-9513-e8ad13c90d81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:46:58.967491 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.967461    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl_3fe75e38-dc7e-42f4-9513-e8ad13c90d81/storage-initializer/1.log"
Sep 17 18:46:58.967938 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.967904    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl_3fe75e38-dc7e-42f4-9513-e8ad13c90d81/storage-initializer/0.log"
Sep 17 18:46:58.967981 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.967937    2561 generic.go:356] "Generic (PLEG): container finished" podID="3fe75e38-dc7e-42f4-9513-e8ad13c90d81" containerID="15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8" exitCode=1
Sep 17 18:46:58.967981 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.967973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl" event={"ID":"3fe75e38-dc7e-42f4-9513-e8ad13c90d81","Type":"ContainerDied","Data":"15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8"}
Sep 17 18:46:58.968042 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.967990    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"
Sep 17 18:46:58.968042 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.968022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl" event={"ID":"3fe75e38-dc7e-42f4-9513-e8ad13c90d81","Type":"ContainerDied","Data":"beea34d6fa52ac8cb5e9139cae5fe88c9870bc74f6baf1bea1085dc124ee5fd1"}
Sep 17 18:46:58.968106 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.968048    2561 scope.go:117] "RemoveContainer" containerID="15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8"
Sep 17 18:46:58.976876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.976854    2561 scope.go:117] "RemoveContainer" containerID="5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc"
Sep 17 18:46:58.984138 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.984121    2561 scope.go:117] "RemoveContainer" containerID="15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8"
Sep 17 18:46:58.984419 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:46:58.984401    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8\": container with ID starting with 15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8 not found: ID does not exist" containerID="15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8"
Sep 17 18:46:58.984466 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.984427    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8"} err="failed to get container status \"15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8\": rpc error: code = NotFound desc = could not find container \"15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8\": container with ID starting with 15ad8c22a656a5fe1f2190e85793dd20f1f91fd66ea27ad1339550bdf4e144b8 not found: ID does not exist"
Sep 17 18:46:58.984466 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.984444    2561 scope.go:117] "RemoveContainer" containerID="5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc"
Sep 17 18:46:58.984720 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:46:58.984703    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc\": container with ID starting with 5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc not found: ID does not exist" containerID="5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc"
Sep 17 18:46:58.984759 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:58.984725    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc"} err="failed to get container status \"5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc\": rpc error: code = NotFound desc = could not find container \"5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc\": container with ID starting with 5537fa0c9c6ad4411e697819d85488a3cbba4131bfac0f31b93c7f41cf694ffc not found: ID does not exist"
Sep 17 18:46:59.010143 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:59.010110    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"]
Sep 17 18:46:59.016012 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:59.015985    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d7124a-predictor-7b864df6c-kv2kl"]
Sep 17 18:46:59.217418 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:46:59.217390    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe75e38-dc7e-42f4-9513-e8ad13c90d81" path="/var/lib/kubelet/pods/3fe75e38-dc7e-42f4-9513-e8ad13c90d81/volumes"
Sep 17 18:47:03.011527 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.011495    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:03.049653 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.049620    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"]
Sep 17 18:47:03.049867 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.049854    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe75e38-dc7e-42f4-9513-e8ad13c90d81" containerName="storage-initializer"
Sep 17 18:47:03.049932 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.049868    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe75e38-dc7e-42f4-9513-e8ad13c90d81" containerName="storage-initializer"
Sep 17 18:47:03.049932 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.049880    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe75e38-dc7e-42f4-9513-e8ad13c90d81" containerName="storage-initializer"
Sep 17 18:47:03.049932 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.049885    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe75e38-dc7e-42f4-9513-e8ad13c90d81" containerName="storage-initializer"
Sep 17 18:47:03.049932 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.049927    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe75e38-dc7e-42f4-9513-e8ad13c90d81" containerName="storage-initializer"
Sep 17 18:47:03.050066 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.050004    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe75e38-dc7e-42f4-9513-e8ad13c90d81" containerName="storage-initializer"
Sep 17 18:47:03.054730 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.054714    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.057398 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.057378    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:47:03.057572 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.057495    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:47:03.057572 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.057546    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:47:03.058530 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.058508    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-9405f-kube-rbac-proxy-sar-config\""
Sep 17 18:47:03.058643 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.058547    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-9405f-predictor-serving-cert\""
Sep 17 18:47:03.058643 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.058613    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\""
Sep 17 18:47:03.058768 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.058713    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:47:03.066651 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.066630    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"]
Sep 17 18:47:03.093307 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.093282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:03.167525 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.167474    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kserve-provision-location\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.167649 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.167537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-cabundle-cert\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.167649 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.167576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b57w\" (UniqueName: \"kubernetes.io/projected/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-api-access-4b57w\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.167649 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.167592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-proxy-tls\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.167763 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.167654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.268555 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.268453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b57w\" (UniqueName: \"kubernetes.io/projected/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-api-access-4b57w\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.268555 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.268508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-proxy-tls\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.268555 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.268538    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.268767 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.268569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kserve-provision-location\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.268767 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.268588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-cabundle-cert\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.269024 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.269003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kserve-provision-location\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.269280 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.269249    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.269280 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.269267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-cabundle-cert\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.271189 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.271168    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-proxy-tls\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.277205 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.277176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b57w\" (UniqueName: \"kubernetes.io/projected/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-api-access-4b57w\") pod \"raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") " pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.364239 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.364199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:03.634228 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.634199    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"]
Sep 17 18:47:03.637095 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:47:03.637068    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58cb97c_687f_4887_a2e6_0032bdbe8a5d.slice/crio-4700cd6817bfaeb5ab333938066323cdbb7e984e5616dc1d70274eee23787243 WatchSource:0}: Error finding container 4700cd6817bfaeb5ab333938066323cdbb7e984e5616dc1d70274eee23787243: Status 404 returned error can't find the container with id 4700cd6817bfaeb5ab333938066323cdbb7e984e5616dc1d70274eee23787243
Sep 17 18:47:03.989258 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.989182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" event={"ID":"a58cb97c-687f-4887-a2e6-0032bdbe8a5d","Type":"ContainerStarted","Data":"b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642"}
Sep 17 18:47:03.989258 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:03.989246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" event={"ID":"a58cb97c-687f-4887-a2e6-0032bdbe8a5d","Type":"ContainerStarted","Data":"4700cd6817bfaeb5ab333938066323cdbb7e984e5616dc1d70274eee23787243"}
Sep 17 18:47:06.767649 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:06.767611    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:06.865741 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:06.865707    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:07.000739 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:07.000705    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:07.211060 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:07.211025    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:08.007867 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:08.007831    2561 generic.go:356] "Generic (PLEG): container finished" podID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerID="b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642" exitCode=0
Sep 17 18:47:08.008301 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:08.007902    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" event={"ID":"a58cb97c-687f-4887-a2e6-0032bdbe8a5d","Type":"ContainerDied","Data":"b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642"}
Sep 17 18:47:09.013909 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:09.013873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" event={"ID":"a58cb97c-687f-4887-a2e6-0032bdbe8a5d","Type":"ContainerStarted","Data":"fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2"}
Sep 17 18:47:09.014306 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:09.013935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" event={"ID":"a58cb97c-687f-4887-a2e6-0032bdbe8a5d","Type":"ContainerStarted","Data":"5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6"}
Sep 17 18:47:09.014306 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:09.014192    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:09.014416 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:09.014328    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:09.015533 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:09.015506    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 18:47:09.038094 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:09.038039    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podStartSLOduration=2.29080631 podStartE2EDuration="6.038023673s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.747217363s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:09.036663213 +0000 UTC m=+1056.316320242" watchObservedRunningTime="2026-09-17 18:47:09.038023673 +0000 UTC m=+1056.317680701"
Sep 17 18:47:10.017713 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:10.017675    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 18:47:15.022570 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:15.022539    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:47:15.023036 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:15.023007    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 18:47:25.023296 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:25.023252    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 18:47:28.391518 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:28.391469    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:31.149577 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:31.149535    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:35.023038 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:35.023003    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 18:47:45.023344 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:45.023300    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 18:47:55.023105 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:47:55.023068    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 18:48:05.022981 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:05.022943    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 18:48:15.023590 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:15.023548    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 18:48:24.213690 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:24.213658    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:48:33.203782 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:33.203737    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"]
Sep 17 18:48:33.204305 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:33.203980    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" containerID="cri-o://5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6" gracePeriod=30
Sep 17 18:48:33.204305 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:33.204040    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kube-rbac-proxy" containerID="cri-o://fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2" gracePeriod=30
Sep 17 18:48:33.216501 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:33.216461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:34.213766 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:34.213727    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 18:48:34.310051 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:34.310019    2561 generic.go:356] "Generic (PLEG): container finished" podID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerID="fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2" exitCode=2
Sep 17 18:48:34.310178 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:34.310093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" event={"ID":"a58cb97c-687f-4887-a2e6-0032bdbe8a5d","Type":"ContainerDied","Data":"fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2"}
Sep 17 18:48:35.018404 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:35.018364    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 17 18:48:37.747697 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.747670    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:48:37.840839 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.840811    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b57w\" (UniqueName: \"kubernetes.io/projected/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-api-access-4b57w\") pod \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") "
Sep 17 18:48:37.840969 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.840861    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kserve-provision-location\") pod \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") "
Sep 17 18:48:37.840969 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.840895    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-rbac-proxy-sar-config\") pod \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") "
Sep 17 18:48:37.840969 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.840919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-proxy-tls\") pod \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") "
Sep 17 18:48:37.840969 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.840944    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-cabundle-cert\") pod \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\" (UID: \"a58cb97c-687f-4887-a2e6-0032bdbe8a5d\") "
Sep 17 18:48:37.841314 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.841285    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a58cb97c-687f-4887-a2e6-0032bdbe8a5d" (UID: "a58cb97c-687f-4887-a2e6-0032bdbe8a5d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:48:37.841374 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.841298    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a58cb97c-687f-4887-a2e6-0032bdbe8a5d" (UID: "a58cb97c-687f-4887-a2e6-0032bdbe8a5d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:48:37.841374 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.841334    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a58cb97c-687f-4887-a2e6-0032bdbe8a5d" (UID: "a58cb97c-687f-4887-a2e6-0032bdbe8a5d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:48:37.843004 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.842979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a58cb97c-687f-4887-a2e6-0032bdbe8a5d" (UID: "a58cb97c-687f-4887-a2e6-0032bdbe8a5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:48:37.843097 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.843008    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-api-access-4b57w" (OuterVolumeSpecName: "kube-api-access-4b57w") pod "a58cb97c-687f-4887-a2e6-0032bdbe8a5d" (UID: "a58cb97c-687f-4887-a2e6-0032bdbe8a5d"). InnerVolumeSpecName "kube-api-access-4b57w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:48:37.942231 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.942209    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:48:37.942231 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.942230    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:48:37.942358 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.942243    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:48:37.942358 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.942251    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b57w\" (UniqueName: \"kubernetes.io/projected/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kube-api-access-4b57w\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:48:37.942358 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:37.942261    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58cb97c-687f-4887-a2e6-0032bdbe8a5d-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:48:38.325490 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.325447    2561 generic.go:356] "Generic (PLEG): container finished" podID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerID="5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6" exitCode=0
Sep 17 18:48:38.325659 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.325522    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"
Sep 17 18:48:38.325659 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.325522    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" event={"ID":"a58cb97c-687f-4887-a2e6-0032bdbe8a5d","Type":"ContainerDied","Data":"5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6"}
Sep 17 18:48:38.325659 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.325644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp" event={"ID":"a58cb97c-687f-4887-a2e6-0032bdbe8a5d","Type":"ContainerDied","Data":"4700cd6817bfaeb5ab333938066323cdbb7e984e5616dc1d70274eee23787243"}
Sep 17 18:48:38.325782 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.325668    2561 scope.go:117] "RemoveContainer" containerID="fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2"
Sep 17 18:48:38.336250 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.336223    2561 scope.go:117] "RemoveContainer" containerID="5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6"
Sep 17 18:48:38.343237 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.343222    2561 scope.go:117] "RemoveContainer" containerID="b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642"
Sep 17 18:48:38.349841 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.349825    2561 scope.go:117] "RemoveContainer" containerID="fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2"
Sep 17 18:48:38.350086 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:48:38.350066    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2\": container with ID starting with fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2 not found: ID does not exist" containerID="fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2"
Sep 17 18:48:38.350137 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.350093    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2"} err="failed to get container status \"fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2\": rpc error: code = NotFound desc = could not find container \"fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2\": container with ID starting with fc47e2d17232e9b28f2d22b7920f411be623d0e0f5454757d9cb12361c081ee2 not found: ID does not exist"
Sep 17 18:48:38.350137 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.350109    2561 scope.go:117] "RemoveContainer" containerID="5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6"
Sep 17 18:48:38.350347 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:48:38.350330    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6\": container with ID starting with 5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6 not found: ID does not exist" containerID="5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6"
Sep 17 18:48:38.350391 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.350353    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6"} err="failed to get container status \"5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6\": rpc error: code = NotFound desc = could not find container \"5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6\": container with ID starting with 5692553cf9c9d9edff9f9873839a0fea0726abec2804798f4a9d45141b8c39c6 not found: ID does not exist"
Sep 17 18:48:38.350391 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.350370    2561 scope.go:117] "RemoveContainer" containerID="b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642"
Sep 17 18:48:38.350634 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:48:38.350619    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642\": container with ID starting with b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642 not found: ID does not exist" containerID="b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642"
Sep 17 18:48:38.350690 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.350638    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642"} err="failed to get container status \"b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642\": rpc error: code = NotFound desc = could not find container \"b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642\": container with ID starting with b8b1adfc8aa2ccc1e51056750363dd2cbf5da5c2e0cf366b4301c17cc833e642 not found: ID does not exist"
Sep 17 18:48:38.353032 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.353012    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"]
Sep 17 18:48:38.357472 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.357453    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9405f-predictor-57dcd48c4f-gd4lp"]
Sep 17 18:48:38.442011 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.441984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:38.568269 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.568206    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"]
Sep 17 18:48:38.568533 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.568517    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container"
Sep 17 18:48:38.568626 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.568536    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container"
Sep 17 18:48:38.568626 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.568563    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kube-rbac-proxy"
Sep 17 18:48:38.568626 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.568571    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kube-rbac-proxy"
Sep 17 18:48:38.568626 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.568593    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="storage-initializer"
Sep 17 18:48:38.568626 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.568602    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="storage-initializer"
Sep 17 18:48:38.568873 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.568667    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kube-rbac-proxy"
Sep 17 18:48:38.568873 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.568681    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" containerName="kserve-container"
Sep 17 18:48:38.572916 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.572897    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.576245 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.576196    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:48:38.576490 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.576462    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:48:38.576805 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.576781    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-c7c54-kube-rbac-proxy-sar-config\""
Sep 17 18:48:38.577014 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.576999    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\""
Sep 17 18:48:38.577178 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.577161    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-c7c54-predictor-serving-cert\""
Sep 17 18:48:38.577247 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.577166    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:48:38.577309 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.577234    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:48:38.586963 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.586942    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"]
Sep 17 18:48:38.647016 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.646986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26d31fd2-7e84-4781-88b2-bc5d43396987-kserve-provision-location\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.647133 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.647057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5whk\" (UniqueName: \"kubernetes.io/projected/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-api-access-x5whk\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.647133 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.647088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.647133 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.647111    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d31fd2-7e84-4781-88b2-bc5d43396987-proxy-tls\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.647133 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.647131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-cabundle-cert\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.748166 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.748134    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d31fd2-7e84-4781-88b2-bc5d43396987-proxy-tls\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.748166 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.748170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-cabundle-cert\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.748702 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.748207    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26d31fd2-7e84-4781-88b2-bc5d43396987-kserve-provision-location\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.748702 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.748234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5whk\" (UniqueName: \"kubernetes.io/projected/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-api-access-x5whk\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.748702 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.748253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.748702 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.748694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26d31fd2-7e84-4781-88b2-bc5d43396987-kserve-provision-location\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.748922 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.748893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.749078 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.749056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-cabundle-cert\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.750615 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.750597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d31fd2-7e84-4781-88b2-bc5d43396987-proxy-tls\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.758130 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.758101    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:38.759608 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.759589    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5whk\" (UniqueName: \"kubernetes.io/projected/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-api-access-x5whk\") pod \"raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:38.882598 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:38.882531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:39.168894 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:39.168810    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"]
Sep 17 18:48:39.171944 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:48:39.171904    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d31fd2_7e84_4781_88b2_bc5d43396987.slice/crio-15c4f79956be7bcee5042e8912cb2740295a54093d4785fe2861126134b4a475 WatchSource:0}: Error finding container 15c4f79956be7bcee5042e8912cb2740295a54093d4785fe2861126134b4a475: Status 404 returned error can't find the container with id 15c4f79956be7bcee5042e8912cb2740295a54093d4785fe2861126134b4a475
Sep 17 18:48:39.217354 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:39.217328    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58cb97c-687f-4887-a2e6-0032bdbe8a5d" path="/var/lib/kubelet/pods/a58cb97c-687f-4887-a2e6-0032bdbe8a5d/volumes"
Sep 17 18:48:39.331364 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:39.331321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" event={"ID":"26d31fd2-7e84-4781-88b2-bc5d43396987","Type":"ContainerStarted","Data":"aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81"}
Sep 17 18:48:39.331607 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:39.331377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" event={"ID":"26d31fd2-7e84-4781-88b2-bc5d43396987","Type":"ContainerStarted","Data":"15c4f79956be7bcee5042e8912cb2740295a54093d4785fe2861126134b4a475"}
Sep 17 18:48:43.346045 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:43.346013    2561 generic.go:356] "Generic (PLEG): container finished" podID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerID="aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81" exitCode=0
Sep 17 18:48:43.346416 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:43.346058    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" event={"ID":"26d31fd2-7e84-4781-88b2-bc5d43396987","Type":"ContainerDied","Data":"aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81"}
Sep 17 18:48:44.351253 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:44.351218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" event={"ID":"26d31fd2-7e84-4781-88b2-bc5d43396987","Type":"ContainerStarted","Data":"cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a"}
Sep 17 18:48:44.351745 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:44.351288    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" event={"ID":"26d31fd2-7e84-4781-88b2-bc5d43396987","Type":"ContainerStarted","Data":"57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac"}
Sep 17 18:48:44.351745 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:44.351511    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:44.371033 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:44.370975    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podStartSLOduration=2.479065838 podStartE2EDuration="6.370962736s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.891896898s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:44.370768501 +0000 UTC m=+1151.650425541" watchObservedRunningTime="2026-09-17 18:48:44.370962736 +0000 UTC m=+1151.650619838"
Sep 17 18:48:45.355193 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:45.355159    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:45.356645 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:45.356615    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:48:46.358623 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:46.358543    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:48:51.362647 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:51.362620    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:48:51.363069 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:48:51.362997    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:49:01.363811 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:49:01.363775    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:49:11.363567 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:49:11.363526    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:49:21.363387 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:49:21.363346    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:49:31.363622 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:49:31.363583    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:49:35.104069 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:49:35.104046    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:49:35.105586 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:49:35.105559    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:49:41.363878 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:49:41.363836    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:49:51.363062 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:49:51.363017    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:49:52.213147 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:49:52.213109    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:49:57.011618 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:49:57.011581    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:02.140690 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.140651    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:02.183720 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.183689    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"]
Sep 17 18:50:02.186714 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.186696    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.189561 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.189543    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 18:50:02.189653 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.189585    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:50:02.200296 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.200276    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"]
Sep 17 18:50:02.213615 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.213596    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:50:02.277707 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.277670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:02.297340 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.297311    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"]
Sep 17 18:50:02.300681 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.300662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.303663 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.303643    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:50:02.314377 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.314348    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"]
Sep 17 18:50:02.333335 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.333312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.333461 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.333353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897aca12-4573-4cc8-97e2-66945982ad97-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.333461 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.333390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.333461 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.333422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pz4h\" (UniqueName: \"kubernetes.io/projected/897aca12-4573-4cc8-97e2-66945982ad97-kube-api-access-4pz4h\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.333634 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.333528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/897aca12-4573-4cc8-97e2-66945982ad97-proxy-tls\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.393626 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.393559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:02.434347 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.434499 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.434499 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897aca12-4573-4cc8-97e2-66945982ad97-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.434499 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434423    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.434680 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434531    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbjp\" (UniqueName: \"kubernetes.io/projected/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-api-access-xnbjp\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.434680 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pz4h\" (UniqueName: \"kubernetes.io/projected/897aca12-4573-4cc8-97e2-66945982ad97-kube-api-access-4pz4h\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.434680 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.434680 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/897aca12-4573-4cc8-97e2-66945982ad97-proxy-tls\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.434909 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897aca12-4573-4cc8-97e2-66945982ad97-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.434909 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.434909 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.434884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.435077 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.435054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.435226 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.435204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.436948 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.436929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/897aca12-4573-4cc8-97e2-66945982ad97-proxy-tls\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.444594 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.444572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pz4h\" (UniqueName: \"kubernetes.io/projected/897aca12-4573-4cc8-97e2-66945982ad97-kube-api-access-4pz4h\") pod \"isvc-canary-create-v2-predictor-77847bfbb9-b6qc2\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.496147 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.496118    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:02.535755 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.535720    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbjp\" (UniqueName: \"kubernetes.io/projected/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-api-access-xnbjp\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.535926 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.535763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.535926 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.535799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.535926 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.535846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.535926 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.535885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.536337 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.536314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.536685 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.536661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.536748 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.536714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.538220 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.538201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.546055 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.546031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbjp\" (UniqueName: \"kubernetes.io/projected/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-api-access-xnbjp\") pod \"isvc-canary-create-predictor-66d699644d-57q2c\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.549315 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.549294    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:02.611236 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.611199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:02.771016 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.770973    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"]
Sep 17 18:50:02.773660 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:50:02.773631    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897aca12_4573_4cc8_97e2_66945982ad97.slice/crio-a4f62c6af3f30e02309dda63592b5e0611eb92b67f553cbd7bbd7bd83960592b WatchSource:0}: Error finding container a4f62c6af3f30e02309dda63592b5e0611eb92b67f553cbd7bbd7bd83960592b: Status 404 returned error can't find the container with id a4f62c6af3f30e02309dda63592b5e0611eb92b67f553cbd7bbd7bd83960592b
Sep 17 18:50:02.883844 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:02.883758    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"]
Sep 17 18:50:02.886282 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:50:02.886236    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4285fa16_cc7e_42f3_bf74_6bbcf53a5185.slice/crio-99d616912bca45ff7467aec50e4f2d554049377259ea99797c0215e1c48da957 WatchSource:0}: Error finding container 99d616912bca45ff7467aec50e4f2d554049377259ea99797c0215e1c48da957: Status 404 returned error can't find the container with id 99d616912bca45ff7467aec50e4f2d554049377259ea99797c0215e1c48da957
Sep 17 18:50:03.618827 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:03.618788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" event={"ID":"4285fa16-cc7e-42f3-bf74-6bbcf53a5185","Type":"ContainerStarted","Data":"cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4"}
Sep 17 18:50:03.619288 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:03.618847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" event={"ID":"4285fa16-cc7e-42f3-bf74-6bbcf53a5185","Type":"ContainerStarted","Data":"99d616912bca45ff7467aec50e4f2d554049377259ea99797c0215e1c48da957"}
Sep 17 18:50:03.620327 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:03.620307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" event={"ID":"897aca12-4573-4cc8-97e2-66945982ad97","Type":"ContainerStarted","Data":"b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f"}
Sep 17 18:50:03.620431 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:03.620353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" event={"ID":"897aca12-4573-4cc8-97e2-66945982ad97","Type":"ContainerStarted","Data":"a4f62c6af3f30e02309dda63592b5e0611eb92b67f553cbd7bbd7bd83960592b"}
Sep 17 18:50:07.635348 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:07.635313    2561 generic.go:356] "Generic (PLEG): container finished" podID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerID="cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4" exitCode=0
Sep 17 18:50:07.635806 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:07.635389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" event={"ID":"4285fa16-cc7e-42f3-bf74-6bbcf53a5185","Type":"ContainerDied","Data":"cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4"}
Sep 17 18:50:07.637049 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:07.637004    2561 generic.go:356] "Generic (PLEG): container finished" podID="897aca12-4573-4cc8-97e2-66945982ad97" containerID="b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f" exitCode=0
Sep 17 18:50:07.637098 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:07.637068    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" event={"ID":"897aca12-4573-4cc8-97e2-66945982ad97","Type":"ContainerDied","Data":"b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f"}
Sep 17 18:50:08.446797 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.446764    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"]
Sep 17 18:50:08.447016 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.446992    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" containerID="cri-o://57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac" gracePeriod=30
Sep 17 18:50:08.447142 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.447082    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kube-rbac-proxy" containerID="cri-o://cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a" gracePeriod=30
Sep 17 18:50:08.478050 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.478018    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:08.642427 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.642391    2561 generic.go:356] "Generic (PLEG): container finished" podID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerID="cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a" exitCode=2
Sep 17 18:50:08.642833 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.642465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" event={"ID":"26d31fd2-7e84-4781-88b2-bc5d43396987","Type":"ContainerDied","Data":"cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a"}
Sep 17 18:50:08.644501 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.644455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" event={"ID":"4285fa16-cc7e-42f3-bf74-6bbcf53a5185","Type":"ContainerStarted","Data":"2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4"}
Sep 17 18:50:08.644638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.644547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" event={"ID":"4285fa16-cc7e-42f3-bf74-6bbcf53a5185","Type":"ContainerStarted","Data":"7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075"}
Sep 17 18:50:08.644886 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.644861    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:08.646973 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.646953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" event={"ID":"897aca12-4573-4cc8-97e2-66945982ad97","Type":"ContainerStarted","Data":"04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb"}
Sep 17 18:50:08.647095 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.646997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" event={"ID":"897aca12-4573-4cc8-97e2-66945982ad97","Type":"ContainerStarted","Data":"a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f"}
Sep 17 18:50:08.647276 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.647256    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:08.647354 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.647312    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:08.648807 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.648776    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:50:08.667527 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.667437    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podStartSLOduration=2.975735107 podStartE2EDuration="6.667418017s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.69168291s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:08.666566751 +0000 UTC m=+1235.946223784" watchObservedRunningTime="2026-09-17 18:50:08.667418017 +0000 UTC m=+1235.947075046"
Sep 17 18:50:08.688722 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:08.688674    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podStartSLOduration=2.683989983 podStartE2EDuration="6.688656375s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.004666392s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:08.687558524 +0000 UTC m=+1235.967215552" watchObservedRunningTime="2026-09-17 18:50:08.688656375 +0000 UTC m=+1235.968313405"
Sep 17 18:50:09.652254 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:09.652169    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:09.652254 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:09.652199    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:50:09.653106 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:09.653066    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:50:10.656560 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:10.656520    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:50:11.359805 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:11.359760    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 18:50:12.213604 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:12.213565    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 18:50:13.185988 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.185964    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:50:13.321529 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.321432    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d31fd2-7e84-4781-88b2-bc5d43396987-proxy-tls\") pod \"26d31fd2-7e84-4781-88b2-bc5d43396987\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") "
Sep 17 18:50:13.321529 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.321475    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26d31fd2-7e84-4781-88b2-bc5d43396987-kserve-provision-location\") pod \"26d31fd2-7e84-4781-88b2-bc5d43396987\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") "
Sep 17 18:50:13.321529 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.321517    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5whk\" (UniqueName: \"kubernetes.io/projected/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-api-access-x5whk\") pod \"26d31fd2-7e84-4781-88b2-bc5d43396987\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") "
Sep 17 18:50:13.322050 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.321617    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-cabundle-cert\") pod \"26d31fd2-7e84-4781-88b2-bc5d43396987\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") "
Sep 17 18:50:13.322050 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.321699    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-rbac-proxy-sar-config\") pod \"26d31fd2-7e84-4781-88b2-bc5d43396987\" (UID: \"26d31fd2-7e84-4781-88b2-bc5d43396987\") "
Sep 17 18:50:13.322050 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.321811    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d31fd2-7e84-4781-88b2-bc5d43396987-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26d31fd2-7e84-4781-88b2-bc5d43396987" (UID: "26d31fd2-7e84-4781-88b2-bc5d43396987"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:50:13.322050 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.322020    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26d31fd2-7e84-4781-88b2-bc5d43396987" (UID: "26d31fd2-7e84-4781-88b2-bc5d43396987"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:50:13.322050 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.322043    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26d31fd2-7e84-4781-88b2-bc5d43396987-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:50:13.322208 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.322079    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26d31fd2-7e84-4781-88b2-bc5d43396987" (UID: "26d31fd2-7e84-4781-88b2-bc5d43396987"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:50:13.323593 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.323562    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26d31fd2-7e84-4781-88b2-bc5d43396987-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26d31fd2-7e84-4781-88b2-bc5d43396987" (UID: "26d31fd2-7e84-4781-88b2-bc5d43396987"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:50:13.323840 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.323817    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-api-access-x5whk" (OuterVolumeSpecName: "kube-api-access-x5whk") pod "26d31fd2-7e84-4781-88b2-bc5d43396987" (UID: "26d31fd2-7e84-4781-88b2-bc5d43396987"). InnerVolumeSpecName "kube-api-access-x5whk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:50:13.422957 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.422924    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d31fd2-7e84-4781-88b2-bc5d43396987-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:50:13.422957 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.422950    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5whk\" (UniqueName: \"kubernetes.io/projected/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-api-access-x5whk\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:50:13.422957 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.422959    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:50:13.423149 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.422968    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26d31fd2-7e84-4781-88b2-bc5d43396987-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:50:13.670340 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.670259    2561 generic.go:356] "Generic (PLEG): container finished" podID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerID="57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac" exitCode=0
Sep 17 18:50:13.670340 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.670313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" event={"ID":"26d31fd2-7e84-4781-88b2-bc5d43396987","Type":"ContainerDied","Data":"57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac"}
Sep 17 18:50:13.670340 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.670318    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"
Sep 17 18:50:13.670610 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.670372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj" event={"ID":"26d31fd2-7e84-4781-88b2-bc5d43396987","Type":"ContainerDied","Data":"15c4f79956be7bcee5042e8912cb2740295a54093d4785fe2861126134b4a475"}
Sep 17 18:50:13.670610 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.670398    2561 scope.go:117] "RemoveContainer" containerID="cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a"
Sep 17 18:50:13.678977 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.678959    2561 scope.go:117] "RemoveContainer" containerID="57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac"
Sep 17 18:50:13.685973 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.685959    2561 scope.go:117] "RemoveContainer" containerID="aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81"
Sep 17 18:50:13.692641 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.692618    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"]
Sep 17 18:50:13.693165 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.693149    2561 scope.go:117] "RemoveContainer" containerID="cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a"
Sep 17 18:50:13.693424 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:50:13.693406    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a\": container with ID starting with cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a not found: ID does not exist" containerID="cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a"
Sep 17 18:50:13.693501 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.693433    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a"} err="failed to get container status \"cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a\": rpc error: code = NotFound desc = could not find container \"cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a\": container with ID starting with cb48a448632b45325f27d241c596fe24eddc36fa66bdc56fa38ddb9ff84e7f9a not found: ID does not exist"
Sep 17 18:50:13.693501 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.693449    2561 scope.go:117] "RemoveContainer" containerID="57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac"
Sep 17 18:50:13.693693 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:50:13.693678    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac\": container with ID starting with 57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac not found: ID does not exist" containerID="57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac"
Sep 17 18:50:13.693733 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.693696    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac"} err="failed to get container status \"57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac\": rpc error: code = NotFound desc = could not find container \"57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac\": container with ID starting with 57850a11c1544a96eb4b473fe626ce88621a9e2065b0dfdf16bbe587e7256eac not found: ID does not exist"
Sep 17 18:50:13.693733 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.693706    2561 scope.go:117] "RemoveContainer" containerID="aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81"
Sep 17 18:50:13.693903 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:50:13.693888    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81\": container with ID starting with aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81 not found: ID does not exist" containerID="aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81"
Sep 17 18:50:13.693945 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.693904    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81"} err="failed to get container status \"aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81\": rpc error: code = NotFound desc = could not find container \"aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81\": container with ID starting with aa80948f0773533c512156958b4fb25471cd09ab6b6d832c70435e3e2d83ef81 not found: ID does not exist"
Sep 17 18:50:13.696666 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:13.696647    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c7c54-predictor-5b55c8d76f-bjvdj"]
Sep 17 18:50:14.656685 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:14.656658    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:50:14.657192 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:14.657159    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:50:15.217397 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:15.217359    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" path="/var/lib/kubelet/pods/26d31fd2-7e84-4781-88b2-bc5d43396987/volumes"
Sep 17 18:50:15.661669 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:15.661645    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:50:15.662214 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:15.662187    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:50:24.657395 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:24.657358    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:50:25.662929 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:25.662895    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:50:34.657387 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:34.657349    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:50:35.662177 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:35.662138    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:50:44.657195 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:44.657158    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:50:45.663077 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:45.663039    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:50:54.657852 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:54.657810    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:50:55.662963 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:50:55.662918    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:51:04.657936 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:04.657898    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:51:05.662091 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:05.662039    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:51:14.657078 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:14.657041    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:51:15.662877 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:15.662837    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:51:24.658411 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:24.658371    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:51:25.663580 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:25.663554    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:51:32.266465 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.266431    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"]
Sep 17 18:51:32.266897 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.266723    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" containerID="cri-o://7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075" gracePeriod=30
Sep 17 18:51:32.266897 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.266767    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kube-rbac-proxy" containerID="cri-o://2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4" gracePeriod=30
Sep 17 18:51:32.274070 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.274047    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"]
Sep 17 18:51:32.274282 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.274257    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" containerID="cri-o://a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f" gracePeriod=30
Sep 17 18:51:32.274401 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.274339    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kube-rbac-proxy" containerID="cri-o://04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb" gracePeriod=30
Sep 17 18:51:32.293328 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.293303    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:32.389864 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.389824    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:32.948705 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.948674    2561 generic.go:356] "Generic (PLEG): container finished" podID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerID="2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4" exitCode=2
Sep 17 18:51:32.948870 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.948751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" event={"ID":"4285fa16-cc7e-42f3-bf74-6bbcf53a5185","Type":"ContainerDied","Data":"2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4"}
Sep 17 18:51:32.950945 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.950919    2561 generic.go:356] "Generic (PLEG): container finished" podID="897aca12-4573-4cc8-97e2-66945982ad97" containerID="04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb" exitCode=2
Sep 17 18:51:32.951080 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:32.950982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" event={"ID":"897aca12-4573-4cc8-97e2-66945982ad97","Type":"ContainerDied","Data":"04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb"}
Sep 17 18:51:34.652658 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:34.652611    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 17 18:51:34.658032 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:34.657998    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 18:51:35.657673 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:35.657635    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 18:51:35.663034 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:35.663010    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 18:51:36.821009 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.820985    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:51:36.912573 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.912544    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kserve-provision-location\") pod \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") "
Sep 17 18:51:36.912698 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.912628    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-rbac-proxy-sar-config\") pod \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") "
Sep 17 18:51:36.912698 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.912689    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-proxy-tls\") pod \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") "
Sep 17 18:51:36.912813 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.912721    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnbjp\" (UniqueName: \"kubernetes.io/projected/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-api-access-xnbjp\") pod \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") "
Sep 17 18:51:36.912813 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.912741    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-cabundle-cert\") pod \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\" (UID: \"4285fa16-cc7e-42f3-bf74-6bbcf53a5185\") "
Sep 17 18:51:36.912921 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.912855    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4285fa16-cc7e-42f3-bf74-6bbcf53a5185" (UID: "4285fa16-cc7e-42f3-bf74-6bbcf53a5185"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:51:36.913180 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.913155    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4285fa16-cc7e-42f3-bf74-6bbcf53a5185" (UID: "4285fa16-cc7e-42f3-bf74-6bbcf53a5185"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:36.913267 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.913210    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4285fa16-cc7e-42f3-bf74-6bbcf53a5185" (UID: "4285fa16-cc7e-42f3-bf74-6bbcf53a5185"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:36.914880 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.914857    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-api-access-xnbjp" (OuterVolumeSpecName: "kube-api-access-xnbjp") pod "4285fa16-cc7e-42f3-bf74-6bbcf53a5185" (UID: "4285fa16-cc7e-42f3-bf74-6bbcf53a5185"). InnerVolumeSpecName "kube-api-access-xnbjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:51:36.914956 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.914886    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4285fa16-cc7e-42f3-bf74-6bbcf53a5185" (UID: "4285fa16-cc7e-42f3-bf74-6bbcf53a5185"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:51:36.915761 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.915745    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:51:36.967701 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.967672    2561 generic.go:356] "Generic (PLEG): container finished" podID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerID="7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075" exitCode=0
Sep 17 18:51:36.967831 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.967754    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"
Sep 17 18:51:36.967831 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.967750    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" event={"ID":"4285fa16-cc7e-42f3-bf74-6bbcf53a5185","Type":"ContainerDied","Data":"7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075"}
Sep 17 18:51:36.967937 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.967922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c" event={"ID":"4285fa16-cc7e-42f3-bf74-6bbcf53a5185","Type":"ContainerDied","Data":"99d616912bca45ff7467aec50e4f2d554049377259ea99797c0215e1c48da957"}
Sep 17 18:51:36.967977 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.967963    2561 scope.go:117] "RemoveContainer" containerID="2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4"
Sep 17 18:51:36.969785 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.969764    2561 generic.go:356] "Generic (PLEG): container finished" podID="897aca12-4573-4cc8-97e2-66945982ad97" containerID="a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f" exitCode=0
Sep 17 18:51:36.969893 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.969833    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"
Sep 17 18:51:36.969893 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.969845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" event={"ID":"897aca12-4573-4cc8-97e2-66945982ad97","Type":"ContainerDied","Data":"a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f"}
Sep 17 18:51:36.969999 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.969914    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2" event={"ID":"897aca12-4573-4cc8-97e2-66945982ad97","Type":"ContainerDied","Data":"a4f62c6af3f30e02309dda63592b5e0611eb92b67f553cbd7bbd7bd83960592b"}
Sep 17 18:51:36.976260 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.976243    2561 scope.go:117] "RemoveContainer" containerID="7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075"
Sep 17 18:51:36.983435 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.983418    2561 scope.go:117] "RemoveContainer" containerID="cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4"
Sep 17 18:51:36.990205 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.990191    2561 scope.go:117] "RemoveContainer" containerID="2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4"
Sep 17 18:51:36.990468 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:51:36.990448    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4\": container with ID starting with 2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4 not found: ID does not exist" containerID="2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4"
Sep 17 18:51:36.990542 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.990493    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4"} err="failed to get container status \"2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4\": rpc error: code = NotFound desc = could not find container \"2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4\": container with ID starting with 2e3d43f1e1399cd261bdaebc74244ea57fb90129fd2ccac369be42f623928fc4 not found: ID does not exist"
Sep 17 18:51:36.990542 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.990517    2561 scope.go:117] "RemoveContainer" containerID="7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075"
Sep 17 18:51:36.990802 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:51:36.990784    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075\": container with ID starting with 7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075 not found: ID does not exist" containerID="7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075"
Sep 17 18:51:36.990854 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.990809    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075"} err="failed to get container status \"7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075\": rpc error: code = NotFound desc = could not find container \"7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075\": container with ID starting with 7bc8c73c7d0b2d21cd60e8934cd1f88e77996605e9b002635d92124a8e173075 not found: ID does not exist"
Sep 17 18:51:36.990854 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.990849    2561 scope.go:117] "RemoveContainer" containerID="cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4"
Sep 17 18:51:36.991099 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:51:36.991084    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4\": container with ID starting with cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4 not found: ID does not exist" containerID="cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4"
Sep 17 18:51:36.991135 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.991102    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4"} err="failed to get container status \"cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4\": rpc error: code = NotFound desc = could not find container \"cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4\": container with ID starting with cc558071df26f77903daaff9065e7ddfe288f97c9656b6c76521961fa121daf4 not found: ID does not exist"
Sep 17 18:51:36.991135 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.991115    2561 scope.go:117] "RemoveContainer" containerID="04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb"
Sep 17 18:51:36.991378 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.991360    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"]
Sep 17 18:51:36.997947 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.997926    2561 scope.go:117] "RemoveContainer" containerID="a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f"
Sep 17 18:51:36.998138 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:36.998114    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-57q2c"]
Sep 17 18:51:37.004347 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.004332    2561 scope.go:117] "RemoveContainer" containerID="b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f"
Sep 17 18:51:37.010768 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.010757    2561 scope.go:117] "RemoveContainer" containerID="04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb"
Sep 17 18:51:37.011021 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:51:37.011006    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb\": container with ID starting with 04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb not found: ID does not exist" containerID="04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb"
Sep 17 18:51:37.011093 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.011025    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb"} err="failed to get container status \"04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb\": rpc error: code = NotFound desc = could not find container \"04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb\": container with ID starting with 04f56a375bc48abe8e783fcfcba714a831772c704aca6ff77c74cf42d3ef4adb not found: ID does not exist"
Sep 17 18:51:37.011093 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.011052    2561 scope.go:117] "RemoveContainer" containerID="a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f"
Sep 17 18:51:37.011272 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:51:37.011254    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f\": container with ID starting with a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f not found: ID does not exist" containerID="a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f"
Sep 17 18:51:37.011312 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.011275    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f"} err="failed to get container status \"a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f\": rpc error: code = NotFound desc = could not find container \"a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f\": container with ID starting with a4fb2d1460ba44cc14b8c93a81dbc5d7f2fc21c66d7c87c8ab632de79b36151f not found: ID does not exist"
Sep 17 18:51:37.011312 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.011287    2561 scope.go:117] "RemoveContainer" containerID="b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f"
Sep 17 18:51:37.011565 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:51:37.011546    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f\": container with ID starting with b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f not found: ID does not exist" containerID="b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f"
Sep 17 18:51:37.011611 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.011570    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f"} err="failed to get container status \"b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f\": rpc error: code = NotFound desc = could not find container \"b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f\": container with ID starting with b8d3fe9f18c42f842d8fb172f43daad62eb13bcedbff407407a75428175b663f not found: ID does not exist"
Sep 17 18:51:37.013799 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.013782    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-cabundle-cert\") pod \"897aca12-4573-4cc8-97e2-66945982ad97\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") "
Sep 17 18:51:37.013854 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.013810    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pz4h\" (UniqueName: \"kubernetes.io/projected/897aca12-4573-4cc8-97e2-66945982ad97-kube-api-access-4pz4h\") pod \"897aca12-4573-4cc8-97e2-66945982ad97\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") "
Sep 17 18:51:37.013854 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.013848    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/897aca12-4573-4cc8-97e2-66945982ad97-proxy-tls\") pod \"897aca12-4573-4cc8-97e2-66945982ad97\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") "
Sep 17 18:51:37.013928 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.013867    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897aca12-4573-4cc8-97e2-66945982ad97-kserve-provision-location\") pod \"897aca12-4573-4cc8-97e2-66945982ad97\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") "
Sep 17 18:51:37.013928 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.013917    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-kube-rbac-proxy-sar-config\") pod \"897aca12-4573-4cc8-97e2-66945982ad97\" (UID: \"897aca12-4573-4cc8-97e2-66945982ad97\") "
Sep 17 18:51:37.014066 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.014051    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:51:37.014113 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.014074    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbjp\" (UniqueName: \"kubernetes.io/projected/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-api-access-xnbjp\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:51:37.014113 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.014097    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:51:37.014113 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.014110    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:51:37.014231 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.014124    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4285fa16-cc7e-42f3-bf74-6bbcf53a5185-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:51:37.014231 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.014130    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "897aca12-4573-4cc8-97e2-66945982ad97" (UID: "897aca12-4573-4cc8-97e2-66945982ad97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:37.014231 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.014209    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897aca12-4573-4cc8-97e2-66945982ad97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "897aca12-4573-4cc8-97e2-66945982ad97" (UID: "897aca12-4573-4cc8-97e2-66945982ad97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:51:37.014335 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.014268    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "897aca12-4573-4cc8-97e2-66945982ad97" (UID: "897aca12-4573-4cc8-97e2-66945982ad97"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:37.015715 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.015695    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897aca12-4573-4cc8-97e2-66945982ad97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "897aca12-4573-4cc8-97e2-66945982ad97" (UID: "897aca12-4573-4cc8-97e2-66945982ad97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:51:37.015800 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.015787    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897aca12-4573-4cc8-97e2-66945982ad97-kube-api-access-4pz4h" (OuterVolumeSpecName: "kube-api-access-4pz4h") pod "897aca12-4573-4cc8-97e2-66945982ad97" (UID: "897aca12-4573-4cc8-97e2-66945982ad97"). InnerVolumeSpecName "kube-api-access-4pz4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:51:37.115276 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.115238    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:51:37.115276 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.115272    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897aca12-4573-4cc8-97e2-66945982ad97-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:51:37.115276 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.115282    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pz4h\" (UniqueName: \"kubernetes.io/projected/897aca12-4573-4cc8-97e2-66945982ad97-kube-api-access-4pz4h\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:51:37.115444 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.115291    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/897aca12-4573-4cc8-97e2-66945982ad97-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:51:37.115444 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.115301    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897aca12-4573-4cc8-97e2-66945982ad97-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:51:37.217205 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.217140    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" path="/var/lib/kubelet/pods/4285fa16-cc7e-42f3-bf74-6bbcf53a5185/volumes"
Sep 17 18:51:37.286137 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.286114    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"]
Sep 17 18:51:37.290026 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.290006    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-77847bfbb9-b6qc2"]
Sep 17 18:51:37.398917 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.398883    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:37.459115 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459090    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"]
Sep 17 18:51:37.459317 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459305    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="storage-initializer"
Sep 17 18:51:37.459364 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459318    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="storage-initializer"
Sep 17 18:51:37.459364 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459332    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kube-rbac-proxy"
Sep 17 18:51:37.459364 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459337    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kube-rbac-proxy"
Sep 17 18:51:37.459364 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459346    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="storage-initializer"
Sep 17 18:51:37.459364 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459352    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="storage-initializer"
Sep 17 18:51:37.459364 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459358    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kube-rbac-proxy"
Sep 17 18:51:37.459364 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459363    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kube-rbac-proxy"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459369    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="storage-initializer"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459375    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="storage-initializer"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459380    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459385    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459394    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459399    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459405    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kube-rbac-proxy"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459410    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kube-rbac-proxy"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459420    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459425    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459461    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kube-rbac-proxy"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459468    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kserve-container"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459475    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4285fa16-cc7e-42f3-bf74-6bbcf53a5185" containerName="kube-rbac-proxy"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459496    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kube-rbac-proxy"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459504    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d31fd2-7e84-4781-88b2-bc5d43396987" containerName="kserve-container"
Sep 17 18:51:37.459638 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.459511    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="897aca12-4573-4cc8-97e2-66945982ad97" containerName="kserve-container"
Sep 17 18:51:37.464071 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.464053    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"]
Sep 17 18:51:37.464196 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.464178    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.466531 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.466511    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:51:37.466653 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.466591    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:51:37.466653 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.466609    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:51:37.466905 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.466888    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:51:37.466905 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.466898    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:51:37.467043 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.466934    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:51:37.467043 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.467004    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\""
Sep 17 18:51:37.467147 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.467131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.469370 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.469326    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:51:37.474351 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.474332    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"]
Sep 17 18:51:37.477457 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.477405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"]
Sep 17 18:51:37.498589 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.498571    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:37.598551 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.598530    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:37.619491 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.619459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.619572 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.619506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.619572 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.619538    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.619572 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.619559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5h4\" (UniqueName: \"kubernetes.io/projected/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-api-access-6x5h4\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.619744 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.619576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-484lc\" (UniqueName: \"kubernetes.io/projected/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-api-access-484lc\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.619744 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.619602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd97d9e0-a938-47c4-bb90-37abc2a08357-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.619744 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.619656    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38ac90f0-f9ff-451c-af9c-2032150dda29-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.619744 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.619709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d9e0-a938-47c4-bb90-37abc2a08357-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.619886 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.619756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38ac90f0-f9ff-451c-af9c-2032150dda29-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.619886 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.619786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.698997 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.698969    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:37.720794 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.720745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-484lc\" (UniqueName: \"kubernetes.io/projected/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-api-access-484lc\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.720794 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.720772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd97d9e0-a938-47c4-bb90-37abc2a08357-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.720794 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.720792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38ac90f0-f9ff-451c-af9c-2032150dda29-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.720999 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.720810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d9e0-a938-47c4-bb90-37abc2a08357-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.720999 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.720837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38ac90f0-f9ff-451c-af9c-2032150dda29-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.720999 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.720865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.720999 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.720896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.720999 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.720927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.720999 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.720953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.720999 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.720978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5h4\" (UniqueName: \"kubernetes.io/projected/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-api-access-6x5h4\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.721345 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.721181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd97d9e0-a938-47c4-bb90-37abc2a08357-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.721345 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.721334    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38ac90f0-f9ff-451c-af9c-2032150dda29-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.721761 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.721741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.721837 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.721773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.721837 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.721824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.721919 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.721903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.723839 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.723815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d9e0-a938-47c4-bb90-37abc2a08357-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.723949 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.723929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38ac90f0-f9ff-451c-af9c-2032150dda29-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.729300 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.729279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-484lc\" (UniqueName: \"kubernetes.io/projected/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-api-access-484lc\") pod \"isvc-canary-promote-predictor-dfdfd784d-kwfrb\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:37.729528 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.729507    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5h4\" (UniqueName: \"kubernetes.io/projected/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-api-access-6x5h4\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.776330 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.776301    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:37.782689 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:37.782672    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:38.050499 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:38.050460    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"]
Sep 17 18:51:38.052549 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:51:38.052519    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd97d9e0_a938_47c4_bb90_37abc2a08357.slice/crio-12660a9a8eecf1d740d44e38027177907ae9f4bf47ec98f77da50d91c5c3fb6e WatchSource:0}: Error finding container 12660a9a8eecf1d740d44e38027177907ae9f4bf47ec98f77da50d91c5c3fb6e: Status 404 returned error can't find the container with id 12660a9a8eecf1d740d44e38027177907ae9f4bf47ec98f77da50d91c5c3fb6e
Sep 17 18:51:38.057279 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:38.057257    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"]
Sep 17 18:51:38.059781 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:51:38.059756    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ac90f0_f9ff_451c_af9c_2032150dda29.slice/crio-cb56d27ee7ec3852eee41022a574d7ecf1a69cabd3823cacaa91cd61912243a7 WatchSource:0}: Error finding container cb56d27ee7ec3852eee41022a574d7ecf1a69cabd3823cacaa91cd61912243a7: Status 404 returned error can't find the container with id cb56d27ee7ec3852eee41022a574d7ecf1a69cabd3823cacaa91cd61912243a7
Sep 17 18:51:38.980775 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:38.980735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" event={"ID":"38ac90f0-f9ff-451c-af9c-2032150dda29","Type":"ContainerStarted","Data":"2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2"}
Sep 17 18:51:38.980951 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:38.980797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" event={"ID":"38ac90f0-f9ff-451c-af9c-2032150dda29","Type":"ContainerStarted","Data":"cb56d27ee7ec3852eee41022a574d7ecf1a69cabd3823cacaa91cd61912243a7"}
Sep 17 18:51:38.982207 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:38.982176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" event={"ID":"bd97d9e0-a938-47c4-bb90-37abc2a08357","Type":"ContainerStarted","Data":"532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f"}
Sep 17 18:51:38.982325 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:38.982237    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" event={"ID":"bd97d9e0-a938-47c4-bb90-37abc2a08357","Type":"ContainerStarted","Data":"12660a9a8eecf1d740d44e38027177907ae9f4bf47ec98f77da50d91c5c3fb6e"}
Sep 17 18:51:39.216992 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:39.216957    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="897aca12-4573-4cc8-97e2-66945982ad97" path="/var/lib/kubelet/pods/897aca12-4573-4cc8-97e2-66945982ad97/volumes"
Sep 17 18:51:41.995241 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:41.995154    2561 generic.go:356] "Generic (PLEG): container finished" podID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerID="2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2" exitCode=0
Sep 17 18:51:41.995697 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:41.995238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" event={"ID":"38ac90f0-f9ff-451c-af9c-2032150dda29","Type":"ContainerDied","Data":"2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2"}
Sep 17 18:51:41.996890 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:41.996868    2561 generic.go:356] "Generic (PLEG): container finished" podID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerID="532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f" exitCode=0
Sep 17 18:51:41.996993 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:41.996935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" event={"ID":"bd97d9e0-a938-47c4-bb90-37abc2a08357","Type":"ContainerDied","Data":"532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f"}
Sep 17 18:51:43.003039 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:43.003006    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" event={"ID":"38ac90f0-f9ff-451c-af9c-2032150dda29","Type":"ContainerStarted","Data":"fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11"}
Sep 17 18:51:43.003512 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:43.003079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" event={"ID":"38ac90f0-f9ff-451c-af9c-2032150dda29","Type":"ContainerStarted","Data":"6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322"}
Sep 17 18:51:43.003512 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:43.003382    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:43.003649 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:43.003527    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:43.005007 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:43.004977    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:51:43.005375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:43.005357    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" event={"ID":"bd97d9e0-a938-47c4-bb90-37abc2a08357","Type":"ContainerStarted","Data":"0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc"}
Sep 17 18:51:43.005445 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:43.005402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" event={"ID":"bd97d9e0-a938-47c4-bb90-37abc2a08357","Type":"ContainerStarted","Data":"b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf"}
Sep 17 18:51:43.005595 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:43.005580    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:43.023148 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:43.023103    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podStartSLOduration=2.462204698 podStartE2EDuration="6.023090063s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.560885365s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:43.021670738 +0000 UTC m=+1330.301327767" watchObservedRunningTime="2026-09-17 18:51:43.023090063 +0000 UTC m=+1330.302747091"
Sep 17 18:51:43.039356 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:43.039308    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podStartSLOduration=2.323518728 podStartE2EDuration="6.039293014s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.715774286s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:43.038430252 +0000 UTC m=+1330.318087281" watchObservedRunningTime="2026-09-17 18:51:43.039293014 +0000 UTC m=+1330.318950043"
Sep 17 18:51:44.010070 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:44.010032    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:51:44.010537 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:44.010140    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:44.011068 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:44.011043    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:51:45.013645 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:45.013605    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:51:49.014804 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:49.014733    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:51:49.015210 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:49.015188    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:51:50.017778 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:50.017753    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:51:50.018217 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:50.018193    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:51:59.016158 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:51:59.016123    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:52:00.018414 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:00.018369    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:52:09.015948 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:09.015909    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:52:10.019143 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:10.019104    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:52:19.015876 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:19.015839    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:52:20.018345 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:20.018307    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:52:29.015781 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:29.015741    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:52:30.019140 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:30.019099    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:52:39.015536 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:39.015495    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:52:40.018456 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:40.018413    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:52:49.015430 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:49.015388    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 18:52:50.018403 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:50.018363    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:52:54.213642 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:54.213611    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:52:54.605369 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:52:54.605341    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:00.019642 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:00.019610    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:53:07.580948 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:07.580916    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"]
Sep 17 18:53:07.581361 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:07.581161    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container" containerID="cri-o://6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322" gracePeriod=30
Sep 17 18:53:07.581361 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:07.581173    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kube-rbac-proxy" containerID="cri-o://fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11" gracePeriod=30
Sep 17 18:53:07.604382 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:07.604352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:08.317642 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:08.317609    2561 generic.go:356] "Generic (PLEG): container finished" podID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerID="fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11" exitCode=2
Sep 17 18:53:08.317814 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:08.317681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" event={"ID":"38ac90f0-f9ff-451c-af9c-2032150dda29","Type":"ContainerDied","Data":"fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11"}
Sep 17 18:53:09.010740 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:09.010698    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 17 18:53:12.229129 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.229101    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:53:12.338831 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.338748    2561 generic.go:356] "Generic (PLEG): container finished" podID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerID="6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322" exitCode=0
Sep 17 18:53:12.338831 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.338812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" event={"ID":"38ac90f0-f9ff-451c-af9c-2032150dda29","Type":"ContainerDied","Data":"6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322"}
Sep 17 18:53:12.338831 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.338822    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"
Sep 17 18:53:12.339027 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.338899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb" event={"ID":"38ac90f0-f9ff-451c-af9c-2032150dda29","Type":"ContainerDied","Data":"cb56d27ee7ec3852eee41022a574d7ecf1a69cabd3823cacaa91cd61912243a7"}
Sep 17 18:53:12.339027 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.338938    2561 scope.go:117] "RemoveContainer" containerID="fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11"
Sep 17 18:53:12.346648 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.346621    2561 scope.go:117] "RemoveContainer" containerID="6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322"
Sep 17 18:53:12.353312 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.353297    2561 scope.go:117] "RemoveContainer" containerID="2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2"
Sep 17 18:53:12.359709 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.359695    2561 scope.go:117] "RemoveContainer" containerID="fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11"
Sep 17 18:53:12.359936 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:53:12.359918    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11\": container with ID starting with fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11 not found: ID does not exist" containerID="fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11"
Sep 17 18:53:12.359984 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.359944    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11"} err="failed to get container status \"fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11\": rpc error: code = NotFound desc = could not find container \"fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11\": container with ID starting with fc42472d9204dd3b6af652c564fcb21edd008f9b84f5d9978282c3aa932ccc11 not found: ID does not exist"
Sep 17 18:53:12.359984 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.359967    2561 scope.go:117] "RemoveContainer" containerID="6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322"
Sep 17 18:53:12.360215 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:53:12.360195    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322\": container with ID starting with 6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322 not found: ID does not exist" containerID="6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322"
Sep 17 18:53:12.360256 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.360222    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322"} err="failed to get container status \"6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322\": rpc error: code = NotFound desc = could not find container \"6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322\": container with ID starting with 6df465ccfb51ea241eb8e7bbfbfcfc13c1e7cdd5ce2a5654eca83981f4cbf322 not found: ID does not exist"
Sep 17 18:53:12.360256 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.360236    2561 scope.go:117] "RemoveContainer" containerID="2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2"
Sep 17 18:53:12.360472 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:53:12.360446    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2\": container with ID starting with 2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2 not found: ID does not exist" containerID="2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2"
Sep 17 18:53:12.360536 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.360493    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2"} err="failed to get container status \"2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2\": rpc error: code = NotFound desc = could not find container \"2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2\": container with ID starting with 2b3a6ad58b23616f52b362a6826e647a076fe544014231dce3d815695176bbc2 not found: ID does not exist"
Sep 17 18:53:12.412253 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.412226    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-rbac-proxy-sar-config\") pod \"38ac90f0-f9ff-451c-af9c-2032150dda29\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") "
Sep 17 18:53:12.412357 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.412257    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-484lc\" (UniqueName: \"kubernetes.io/projected/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-api-access-484lc\") pod \"38ac90f0-f9ff-451c-af9c-2032150dda29\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") "
Sep 17 18:53:12.412357 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.412324    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-cabundle-cert\") pod \"38ac90f0-f9ff-451c-af9c-2032150dda29\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") "
Sep 17 18:53:12.412556 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.412527    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38ac90f0-f9ff-451c-af9c-2032150dda29-proxy-tls\") pod \"38ac90f0-f9ff-451c-af9c-2032150dda29\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") "
Sep 17 18:53:12.412556 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.412543    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "38ac90f0-f9ff-451c-af9c-2032150dda29" (UID: "38ac90f0-f9ff-451c-af9c-2032150dda29"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:12.412663 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.412580    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38ac90f0-f9ff-451c-af9c-2032150dda29-kserve-provision-location\") pod \"38ac90f0-f9ff-451c-af9c-2032150dda29\" (UID: \"38ac90f0-f9ff-451c-af9c-2032150dda29\") "
Sep 17 18:53:12.412663 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.412607    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38ac90f0-f9ff-451c-af9c-2032150dda29" (UID: "38ac90f0-f9ff-451c-af9c-2032150dda29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:12.412767 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.412755    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:53:12.412826 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.412768    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:53:12.412931 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.412911    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ac90f0-f9ff-451c-af9c-2032150dda29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38ac90f0-f9ff-451c-af9c-2032150dda29" (UID: "38ac90f0-f9ff-451c-af9c-2032150dda29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:12.414170 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.414153    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-api-access-484lc" (OuterVolumeSpecName: "kube-api-access-484lc") pod "38ac90f0-f9ff-451c-af9c-2032150dda29" (UID: "38ac90f0-f9ff-451c-af9c-2032150dda29"). InnerVolumeSpecName "kube-api-access-484lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:12.414273 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.414250    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ac90f0-f9ff-451c-af9c-2032150dda29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38ac90f0-f9ff-451c-af9c-2032150dda29" (UID: "38ac90f0-f9ff-451c-af9c-2032150dda29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:12.513736 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.513696    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38ac90f0-f9ff-451c-af9c-2032150dda29-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:53:12.513736 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.513731    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38ac90f0-f9ff-451c-af9c-2032150dda29-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:53:12.513736 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.513740    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-484lc\" (UniqueName: \"kubernetes.io/projected/38ac90f0-f9ff-451c-af9c-2032150dda29-kube-api-access-484lc\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:53:12.660980 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.660949    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"]
Sep 17 18:53:12.665579 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:12.665556    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-kwfrb"]
Sep 17 18:53:13.217498 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:13.217457    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" path="/var/lib/kubelet/pods/38ac90f0-f9ff-451c-af9c-2032150dda29/volumes"
Sep 17 18:53:37.632514 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:53:37.632452    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:35.124167 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:35.124140    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:54:35.126403 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:35.126379    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:54:37.813009 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:37.812977    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"]
Sep 17 18:54:37.813452 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:37.813343    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" containerID="cri-o://b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf" gracePeriod=30
Sep 17 18:54:37.813452 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:37.813393    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kube-rbac-proxy" containerID="cri-o://0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc" gracePeriod=30
Sep 17 18:54:38.639518 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:38.639465    2561 generic.go:356] "Generic (PLEG): container finished" podID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerID="0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc" exitCode=2
Sep 17 18:54:38.639697 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:38.639538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" event={"ID":"bd97d9e0-a938-47c4-bb90-37abc2a08357","Type":"ContainerDied","Data":"0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc"}
Sep 17 18:54:39.157461 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:39.157426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:40.014017 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:40.013975    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 18:54:40.018284 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:40.018259    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 18:54:42.452121 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.452098    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:54:42.528338 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.528274    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d9e0-a938-47c4-bb90-37abc2a08357-proxy-tls\") pod \"bd97d9e0-a938-47c4-bb90-37abc2a08357\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") "
Sep 17 18:54:42.528338 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.528306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-rbac-proxy-sar-config\") pod \"bd97d9e0-a938-47c4-bb90-37abc2a08357\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") "
Sep 17 18:54:42.528338 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.528339    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd97d9e0-a938-47c4-bb90-37abc2a08357-kserve-provision-location\") pod \"bd97d9e0-a938-47c4-bb90-37abc2a08357\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") "
Sep 17 18:54:42.528634 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.528357    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-cabundle-cert\") pod \"bd97d9e0-a938-47c4-bb90-37abc2a08357\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") "
Sep 17 18:54:42.528634 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.528583    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x5h4\" (UniqueName: \"kubernetes.io/projected/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-api-access-6x5h4\") pod \"bd97d9e0-a938-47c4-bb90-37abc2a08357\" (UID: \"bd97d9e0-a938-47c4-bb90-37abc2a08357\") "
Sep 17 18:54:42.528741 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.528651    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd97d9e0-a938-47c4-bb90-37abc2a08357-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd97d9e0-a938-47c4-bb90-37abc2a08357" (UID: "bd97d9e0-a938-47c4-bb90-37abc2a08357"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:42.528786 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.528736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd97d9e0-a938-47c4-bb90-37abc2a08357" (UID: "bd97d9e0-a938-47c4-bb90-37abc2a08357"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:42.528786 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.528756    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd97d9e0-a938-47c4-bb90-37abc2a08357" (UID: "bd97d9e0-a938-47c4-bb90-37abc2a08357"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:42.528858 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.528788    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd97d9e0-a938-47c4-bb90-37abc2a08357-kserve-provision-location\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:54:42.530431 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.530408    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd97d9e0-a938-47c4-bb90-37abc2a08357-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd97d9e0-a938-47c4-bb90-37abc2a08357" (UID: "bd97d9e0-a938-47c4-bb90-37abc2a08357"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:42.530519 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.530464    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-api-access-6x5h4" (OuterVolumeSpecName: "kube-api-access-6x5h4") pod "bd97d9e0-a938-47c4-bb90-37abc2a08357" (UID: "bd97d9e0-a938-47c4-bb90-37abc2a08357"). InnerVolumeSpecName "kube-api-access-6x5h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:42.629653 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.629614    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd97d9e0-a938-47c4-bb90-37abc2a08357-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:54:42.629653 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.629651    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:54:42.629653 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.629661    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd97d9e0-a938-47c4-bb90-37abc2a08357-cabundle-cert\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:54:42.629848 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.629670    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x5h4\" (UniqueName: \"kubernetes.io/projected/bd97d9e0-a938-47c4-bb90-37abc2a08357-kube-api-access-6x5h4\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\""
Sep 17 18:54:42.656169 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.656134    2561 generic.go:356] "Generic (PLEG): container finished" podID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerID="b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf" exitCode=0
Sep 17 18:54:42.656296 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.656197    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"
Sep 17 18:54:42.656296 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.656219    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" event={"ID":"bd97d9e0-a938-47c4-bb90-37abc2a08357","Type":"ContainerDied","Data":"b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf"}
Sep 17 18:54:42.656296 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.656278    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl" event={"ID":"bd97d9e0-a938-47c4-bb90-37abc2a08357","Type":"ContainerDied","Data":"12660a9a8eecf1d740d44e38027177907ae9f4bf47ec98f77da50d91c5c3fb6e"}
Sep 17 18:54:42.656452 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.656305    2561 scope.go:117] "RemoveContainer" containerID="0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc"
Sep 17 18:54:42.666130 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.666111    2561 scope.go:117] "RemoveContainer" containerID="b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf"
Sep 17 18:54:42.673381 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.673365    2561 scope.go:117] "RemoveContainer" containerID="532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f"
Sep 17 18:54:42.680165 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.680147    2561 scope.go:117] "RemoveContainer" containerID="0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc"
Sep 17 18:54:42.680413 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:54:42.680394    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc\": container with ID starting with 0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc not found: ID does not exist" containerID="0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc"
Sep 17 18:54:42.680475 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.680420    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc"} err="failed to get container status \"0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc\": rpc error: code = NotFound desc = could not find container \"0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc\": container with ID starting with 0b2109e2d131fd9b53a90fb95709dcffe193d9379a858755f60b2c90cc097cfc not found: ID does not exist"
Sep 17 18:54:42.680475 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.680436    2561 scope.go:117] "RemoveContainer" containerID="b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf"
Sep 17 18:54:42.680688 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:54:42.680670    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf\": container with ID starting with b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf not found: ID does not exist" containerID="b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf"
Sep 17 18:54:42.680736 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.680692    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf"} err="failed to get container status \"b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf\": rpc error: code = NotFound desc = could not find container \"b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf\": container with ID starting with b40506a5f15886324a4710808d3bbed085e80052d389ac103fba372cdda23dcf not found: ID does not exist"
Sep 17 18:54:42.680736 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.680704    2561 scope.go:117] "RemoveContainer" containerID="532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f"
Sep 17 18:54:42.680906 ip-10-0-138-74 kubenswrapper[2561]: E0917 18:54:42.680889    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f\": container with ID starting with 532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f not found: ID does not exist" containerID="532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f"
Sep 17 18:54:42.680943 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.680909    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f"} err="failed to get container status \"532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f\": rpc error: code = NotFound desc = could not find container \"532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f\": container with ID starting with 532109274abf30f95d7d6dd3715872eb23114b32f29d4158e41150983e2a476f not found: ID does not exist"
Sep 17 18:54:42.683714 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.683692    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"]
Sep 17 18:54:42.689099 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:42.689078    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-sljbl"]
Sep 17 18:54:43.217089 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:43.217059    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" path="/var/lib/kubelet/pods/bd97d9e0-a938-47c4-bb90-37abc2a08357/volumes"
Sep 17 18:54:53.197216 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:53.197179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:53.337302 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:54:53.337271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:03.204321 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:55:03.204286    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:04.545104 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:55:04.545072    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:51.903398 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:55:51.903367    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:38.525423 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:56:38.525385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:38.622160 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:56:38.622126    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:40.191045 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:56:40.191014    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:40.272254 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:56:40.272225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:11.841966 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.841928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:11.984117 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984087    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b6x4n/must-gather-twvzj"]
Sep 17 18:57:11.984327 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984315    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="storage-initializer"
Sep 17 18:57:11.984375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984329    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="storage-initializer"
Sep 17 18:57:11.984375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984339    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kube-rbac-proxy"
Sep 17 18:57:11.984375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984344    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kube-rbac-proxy"
Sep 17 18:57:11.984375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984357    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container"
Sep 17 18:57:11.984375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984363    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container"
Sep 17 18:57:11.984375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984370    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="storage-initializer"
Sep 17 18:57:11.984375 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984375    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="storage-initializer"
Sep 17 18:57:11.984622 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984387    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container"
Sep 17 18:57:11.984622 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984392    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container"
Sep 17 18:57:11.984622 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984398    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kube-rbac-proxy"
Sep 17 18:57:11.984622 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984403    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kube-rbac-proxy"
Sep 17 18:57:11.984622 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984442    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kserve-container"
Sep 17 18:57:11.984622 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984448    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kserve-container"
Sep 17 18:57:11.984622 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984455    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ac90f0-f9ff-451c-af9c-2032150dda29" containerName="kube-rbac-proxy"
Sep 17 18:57:11.984622 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.984461    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd97d9e0-a938-47c4-bb90-37abc2a08357" containerName="kube-rbac-proxy"
Sep 17 18:57:11.987196 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.987181    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6x4n/must-gather-twvzj"
Sep 17 18:57:11.990892 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.990869    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b6x4n\"/\"default-dockercfg-tjq4r\""
Sep 17 18:57:11.990892 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.990884    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b6x4n\"/\"kube-root-ca.crt\""
Sep 17 18:57:11.991149 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.991135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b6x4n\"/\"openshift-service-ca.crt\""
Sep 17 18:57:11.999073 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:11.999051    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6x4n/must-gather-twvzj"]
Sep 17 18:57:12.003615 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.003592    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:12.096686 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.096617    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:12.136972 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.136950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a3788acc-946d-42ad-b397-08a2590589a5-must-gather-output\") pod \"must-gather-twvzj\" (UID: \"a3788acc-946d-42ad-b397-08a2590589a5\") " pod="openshift-must-gather-b6x4n/must-gather-twvzj"
Sep 17 18:57:12.137050 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.136989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhxr\" (UniqueName: \"kubernetes.io/projected/a3788acc-946d-42ad-b397-08a2590589a5-kube-api-access-bfhxr\") pod \"must-gather-twvzj\" (UID: \"a3788acc-946d-42ad-b397-08a2590589a5\") " pod="openshift-must-gather-b6x4n/must-gather-twvzj"
Sep 17 18:57:12.237518 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.237476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a3788acc-946d-42ad-b397-08a2590589a5-must-gather-output\") pod \"must-gather-twvzj\" (UID: \"a3788acc-946d-42ad-b397-08a2590589a5\") " pod="openshift-must-gather-b6x4n/must-gather-twvzj"
Sep 17 18:57:12.237666 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.237537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhxr\" (UniqueName: \"kubernetes.io/projected/a3788acc-946d-42ad-b397-08a2590589a5-kube-api-access-bfhxr\") pod \"must-gather-twvzj\" (UID: \"a3788acc-946d-42ad-b397-08a2590589a5\") " pod="openshift-must-gather-b6x4n/must-gather-twvzj"
Sep 17 18:57:12.237817 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.237798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a3788acc-946d-42ad-b397-08a2590589a5-must-gather-output\") pod \"must-gather-twvzj\" (UID: \"a3788acc-946d-42ad-b397-08a2590589a5\") " pod="openshift-must-gather-b6x4n/must-gather-twvzj"
Sep 17 18:57:12.247376 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.247342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhxr\" (UniqueName: \"kubernetes.io/projected/a3788acc-946d-42ad-b397-08a2590589a5-kube-api-access-bfhxr\") pod \"must-gather-twvzj\" (UID: \"a3788acc-946d-42ad-b397-08a2590589a5\") " pod="openshift-must-gather-b6x4n/must-gather-twvzj"
Sep 17 18:57:12.295727 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.295706    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6x4n/must-gather-twvzj"
Sep 17 18:57:12.568044 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.568017    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6x4n/must-gather-twvzj"]
Sep 17 18:57:12.569965 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:57:12.569930    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3788acc_946d_42ad_b397_08a2590589a5.slice/crio-4a7cef6a8ed5f08e5467a47d06802ef8e81eb224a7d3807754d607964c695384 WatchSource:0}: Error finding container 4a7cef6a8ed5f08e5467a47d06802ef8e81eb224a7d3807754d607964c695384: Status 404 returned error can't find the container with id 4a7cef6a8ed5f08e5467a47d06802ef8e81eb224a7d3807754d607964c695384
Sep 17 18:57:12.571858 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:12.571841    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:57:13.157684 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:13.157651    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6x4n/must-gather-twvzj" event={"ID":"a3788acc-946d-42ad-b397-08a2590589a5","Type":"ContainerStarted","Data":"4a7cef6a8ed5f08e5467a47d06802ef8e81eb224a7d3807754d607964c695384"}
Sep 17 18:57:14.163260 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:14.163227    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6x4n/must-gather-twvzj" event={"ID":"a3788acc-946d-42ad-b397-08a2590589a5","Type":"ContainerStarted","Data":"d6654b67cfeaa19a81b4897aa4a9e8b1e311b0e0783f9d44b4993868b9ec6b4e"}
Sep 17 18:57:14.163670 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:14.163270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6x4n/must-gather-twvzj" event={"ID":"a3788acc-946d-42ad-b397-08a2590589a5","Type":"ContainerStarted","Data":"2dfee8591c81a67eff4491db1aecb1b475d95c4a3619c3d6520b208a5be5ce39"}
Sep 17 18:57:14.182326 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:14.182261    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b6x4n/must-gather-twvzj" podStartSLOduration=2.339708357 podStartE2EDuration="3.182237929s" totalImagesPullingTime="842.529572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:57:14.181272276 +0000 UTC m=+1661.460929306" watchObservedRunningTime="2026-09-17 18:57:14.182237929 +0000 UTC m=+1661.461894960"
Sep 17 18:57:14.887583 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:14.887547    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w4s89_e3704c11-e37f-415e-b1dd-1b0facfeaa42/global-pull-secret-syncer/0.log"
Sep 17 18:57:14.939918 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:14.939890    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log"
Sep 17 18:57:15.074949 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:15.074919    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-74.ec2.internal_467b51f32347639980b9f38cde96784b/haproxy/0.log"
Sep 17 18:57:17.881798 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:17.881735    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:17.959061 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:17.958999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4d7m6_9366a1c0-eed2-4f5b-9a11-ada95d186caf/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:19.343956 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:19.343906    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5nvz6_0f609f65-9e6b-42fc-bc03-2809aed32e44/node-exporter/0.log"
Sep 17 18:57:19.369071 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:19.368986    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5nvz6_0f609f65-9e6b-42fc-bc03-2809aed32e44/kube-rbac-proxy/0.log"
Sep 17 18:57:19.399080 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:19.399054    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5nvz6_0f609f65-9e6b-42fc-bc03-2809aed32e44/init-textfile/0.log"
Sep 17 18:57:23.243263 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.243223    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"]
Sep 17 18:57:23.247159 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.247136    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.263542 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.263513    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"]
Sep 17 18:57:23.449121 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.449079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-proc\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.449374 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.449149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-podres\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.449374 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.449182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfq2x\" (UniqueName: \"kubernetes.io/projected/a5bdb95f-522d-4a70-b1f6-24faf759775d-kube-api-access-lfq2x\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.449374 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.449216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-sys\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.449374 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.449314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-lib-modules\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.550693 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.550647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-proc\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.550888 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.550734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-podres\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.550888 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.550770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfq2x\" (UniqueName: \"kubernetes.io/projected/a5bdb95f-522d-4a70-b1f6-24faf759775d-kube-api-access-lfq2x\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.550888 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.550811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-sys\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.550888 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.550846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-lib-modules\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.551105 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.551035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-lib-modules\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.551170 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.551105    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-proc\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.551226 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.551181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-podres\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.551618 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.551590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5bdb95f-522d-4a70-b1f6-24faf759775d-sys\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.565161 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.565121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfq2x\" (UniqueName: \"kubernetes.io/projected/a5bdb95f-522d-4a70-b1f6-24faf759775d-kube-api-access-lfq2x\") pod \"perf-node-gather-daemonset-2wn5q\" (UID: \"a5bdb95f-522d-4a70-b1f6-24faf759775d\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:23.859427 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:23.859325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:24.152252 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:24.151814    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"]
Sep 17 18:57:24.156537 ip-10-0-138-74 kubenswrapper[2561]: W0917 18:57:24.156471    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda5bdb95f_522d_4a70_b1f6_24faf759775d.slice/crio-6ffd3807ef76c0640bc41e1696cfb098c6aaedb9f13f4d77dafa30560d8307e6 WatchSource:0}: Error finding container 6ffd3807ef76c0640bc41e1696cfb098c6aaedb9f13f4d77dafa30560d8307e6: Status 404 returned error can't find the container with id 6ffd3807ef76c0640bc41e1696cfb098c6aaedb9f13f4d77dafa30560d8307e6
Sep 17 18:57:24.178626 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:24.178603    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c766f_5f89ea57-8bac-4b31-9331-f76a08fed0f4/dns/0.log"
Sep 17 18:57:24.200741 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:24.200670    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c766f_5f89ea57-8bac-4b31-9331-f76a08fed0f4/kube-rbac-proxy/0.log"
Sep 17 18:57:24.214455 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:24.214379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q" event={"ID":"a5bdb95f-522d-4a70-b1f6-24faf759775d","Type":"ContainerStarted","Data":"6ffd3807ef76c0640bc41e1696cfb098c6aaedb9f13f4d77dafa30560d8307e6"}
Sep 17 18:57:24.388282 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:24.388252    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z6jng_97b2971f-a12f-494b-83aa-6c14a77418fd/dns-node-resolver/0.log"
Sep 17 18:57:24.927170 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:24.927143    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xdtn6_22c6c31c-3640-4498-bf6f-4023379af97a/node-ca/0.log"
Sep 17 18:57:25.220205 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:25.220130    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q" event={"ID":"a5bdb95f-522d-4a70-b1f6-24faf759775d","Type":"ContainerStarted","Data":"9106c38c68ad92cef872c43efb0ed6841a6e5f4e425c9f8220fdfcef296026fe"}
Sep 17 18:57:25.220377 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:25.220245    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:25.239208 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:25.239147    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q" podStartSLOduration=2.23912843 podStartE2EDuration="2.23912843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:57:25.237287585 +0000 UTC m=+1672.516944614" watchObservedRunningTime="2026-09-17 18:57:25.23912843 +0000 UTC m=+1672.518785461"
Sep 17 18:57:26.052019 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:26.051989    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6vfht_3aae0230-f1ff-48ca-b9b2-edf0dfe312ed/serve-healthcheck-canary/0.log"
Sep 17 18:57:26.480081 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:26.480013    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cp5ds_6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f/kube-rbac-proxy/0.log"
Sep 17 18:57:26.501760 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:26.501733    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cp5ds_6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f/exporter/0.log"
Sep 17 18:57:26.522367 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:26.522339    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cp5ds_6bbb4d50-7d43-4d97-9ee4-e3202e1bc73f/extractor/0.log"
Sep 17 18:57:29.003517 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:29.003449    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-zbf56_d37fab35-3cd3-479a-a6a4-a6e4e62459da/manager/0.log"
Sep 17 18:57:29.026319 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:29.026299    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-pxkcr_40266cf5-6bf8-4472-a725-af8ab4ffd67c/server/0.log"
Sep 17 18:57:31.235390 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:31.235354    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-2wn5q"
Sep 17 18:57:34.971528 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:34.971493    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8gvdb_7a825fbc-a453-4665-a358-cd2d1b343a3b/kube-multus/0.log"
Sep 17 18:57:35.209535 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:35.209459    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sbr_cdb005e3-9caa-4fd7-ae41-bc8936506eb2/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:57:35.240688 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:35.240651    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sbr_cdb005e3-9caa-4fd7-ae41-bc8936506eb2/egress-router-binary-copy/0.log"
Sep 17 18:57:35.265202 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:35.265176    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sbr_cdb005e3-9caa-4fd7-ae41-bc8936506eb2/cni-plugins/0.log"
Sep 17 18:57:35.289144 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:35.289116    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sbr_cdb005e3-9caa-4fd7-ae41-bc8936506eb2/bond-cni-plugin/0.log"
Sep 17 18:57:35.314557 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:35.314535    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sbr_cdb005e3-9caa-4fd7-ae41-bc8936506eb2/routeoverride-cni/0.log"
Sep 17 18:57:35.344081 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:35.344060    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sbr_cdb005e3-9caa-4fd7-ae41-bc8936506eb2/whereabouts-cni-bincopy/0.log"
Sep 17 18:57:35.381860 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:35.381837    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6sbr_cdb005e3-9caa-4fd7-ae41-bc8936506eb2/whereabouts-cni/0.log"
Sep 17 18:57:35.690197 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:35.690171    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p5hr5_1213f80e-4a5f-4930-ae2e-2734dad1d892/network-metrics-daemon/0.log"
Sep 17 18:57:35.721386 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:35.721362    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p5hr5_1213f80e-4a5f-4930-ae2e-2734dad1d892/kube-rbac-proxy/0.log"
Sep 17 18:57:36.960457 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:36.960430    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-controller/0.log"
Sep 17 18:57:36.985883 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:36.985857    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/1.log"
Sep 17 18:57:36.987197 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:36.987179    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovn-acl-logging/0.log"
Sep 17 18:57:37.008914 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:37.008892    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/kube-rbac-proxy-node/0.log"
Sep 17 18:57:37.040964 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:37.040911    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:57:37.080096 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:37.080071    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/northd/0.log"
Sep 17 18:57:37.106067 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:37.106051    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/nbdb/0.log"
Sep 17 18:57:37.136199 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:37.136182    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/sbdb/0.log"
Sep 17 18:57:37.196034 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:37.196013    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhfjv_30074ef6-2972-4a1d-ad80-fa6f1319747e/ovnkube-controller/0.log"
Sep 17 18:57:38.460172 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:38.460139    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-szc74_be355727-2ed7-493b-9067-832bffebf603/network-check-target-container/0.log"
Sep 17 18:57:39.483002 ip-10-0-138-74 kubenswrapper[2561]: I0917 18:57:39.482977    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zxrrd_022b9b32-4714-42a9-b5f7-542385cc4352/iptables-alerter/0.log"