Jul 28 12:33:32.541027 ip-10-0-138-58 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 12:33:32.870821 ip-10-0-138-58 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:33:32.870821 ip-10-0-138-58 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 12:33:32.870821 ip-10-0-138-58 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:33:32.870821 ip-10-0-138-58 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 12:33:32.870821 ip-10-0-138-58 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:33:32.872801 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.872696    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 12:33:32.878414 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878399    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:32.878414 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878415    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878418    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878421    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878428    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878432    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878434    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878437    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878440    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878443    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878446    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878448    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878451    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878453    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878456    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878459    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878463    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878468    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878471    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878474    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:32.878485 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878476    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878479    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878482    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878485    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878487    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878490    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878493    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878496    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878499    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878502    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878505    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878508    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878510    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878513    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878515    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878518    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878522    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878525    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878527    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878529    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:32.878972 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878532    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878534    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878537    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878540    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878542    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878545    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878547    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878550    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878552    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878555    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878558    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878560    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878563    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878566    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878569    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878573    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878576    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878578    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878581    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:32.879563 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878584    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878586    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878589    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878591    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878595    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878597    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878600    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878602    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878605    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878607    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878609    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878612    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878616    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878620    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878622    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878625    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878627    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878630    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878632    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878635    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:32.880337 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878637    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878640    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878643    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878645    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878648    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878651    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878653    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878656    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878659    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878662    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878664    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878667    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878670    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878672    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878675    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878677    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878680    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878683    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878685    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:32.880893 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.878688    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:32.881978 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.881961    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:32.881978 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.881977    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.881982    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.881988    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.881991    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.881995    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.881998    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882001    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882004    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882007    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882009    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882013    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882015    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882019    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882022    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882025    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882030    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882033    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882036    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882039    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882042    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:32.882068 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882044    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:32.882568 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882047    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:32.882568 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882050    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:32.882568 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882052    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:32.882568 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882055    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:32.882568 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882058    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:32.882568 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.882061    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:32.886811 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886801    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:32.886811 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886809    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:32.886811 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886812    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886815    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886818    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886821    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886824    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886826    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886829    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886832    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886835    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886837    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886841    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886843    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886846    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886848    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886851    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886853    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886856    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886858    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886861    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:32.886911 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886864    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886866    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886871    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886874    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886878    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886881    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886883    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886886    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886889    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886893    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886898    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886907    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886910    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886914    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886917    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886920    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886925    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886928    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886931    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:32.887377 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886935    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886938    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886940    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886943    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886946    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886949    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886953    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886956    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886959    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886961    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886964    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886966    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886969    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886972    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886974    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886978    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886981    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886984    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886987    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886989    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:32.887879 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886994    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886996    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.886999    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887002    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887005    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887008    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887010    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887013    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887016    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887018    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887021    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887405    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887417    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887420    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887423    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887426    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887429    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887431    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887434    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:32.888376 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887438    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887442    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887445    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887447    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887451    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887454    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887458    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887461    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887466    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887469    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887472    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887475    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887478    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887481    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887484    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887487    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887489    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887492    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887495    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887498    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:32.888868 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887501    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887504    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887506    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887509    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887511    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887514    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887517    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887520    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887522    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887525    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887527    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887530    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887532    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887535    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887537    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887541    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:32.889363 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887543    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:32.889793 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.887546    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:32.889793 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889792    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889795    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889799    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889803    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889805    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889808    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889811    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889813    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889816    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889818    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889821    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889824    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889826    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889829    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889832    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889835    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889838    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889840    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889842    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:32.889851 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889845    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889847    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889850    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889853    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889855    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889858    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889862    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889866    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889869    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889872    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889874    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889877    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889880    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889882    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889885    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889887    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889890    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889900    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889903    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:32.890316 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889905    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889908    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889911    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889913    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889916    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889918    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889921    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889923    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889926    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889928    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889931    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889934    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889936    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889938    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.889941    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890162    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890170    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890176    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890181    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890185    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890188    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 12:33:32.890797 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890192    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890196    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890200    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890203    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890206    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890210    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890213    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890216    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890219    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890222    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890226    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890229    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890234    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890236    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890239    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890242    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890246    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890250    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890253    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890256    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890260    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890262    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890266    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890269    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 12:33:32.891406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890272    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890275    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890279    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890282    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890285    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890288    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890291    2561 flags.go:64] FLAG: --enable-server="true"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890294    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890298    2561 flags.go:64] FLAG: --event-burst="100"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890301    2561 flags.go:64] FLAG: --event-qps="50"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890305    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890308    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890310    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890314    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890317    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890320    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890323    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890326    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890330    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890333    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890336    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890339    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890342    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890345    2561 flags.go:64] FLAG: --feature-gates=""
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890348    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 12:33:32.892008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890352    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890355    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890359    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890362    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890365    2561 flags.go:64] FLAG: --help="false"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890368    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-138-58.ec2.internal"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890371    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890374    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890377    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890380    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890384    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890387    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890390    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890393    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890396    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890399    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890402    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890405    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890407    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890410    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890413    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890416    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890419    2561 flags.go:64] FLAG: --lock-file=""
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890422    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 12:33:32.892612 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890425    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890428    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890434    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890437    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890440    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890443    2561 flags.go:64] FLAG: --logging-format="text"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890446    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890449    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890452    2561 flags.go:64] FLAG: --manifest-url=""
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890454    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890459    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890462    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890466    2561 flags.go:64] FLAG: --max-pods="110"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890469    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890472    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890474    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890477    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890480    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890483    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890486    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890493    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890496    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890499    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 12:33:32.893223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890502    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890505    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890510    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890513    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890515    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890518    2561 flags.go:64] FLAG: --port="10250"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890521    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890524    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03e5b85a9896e406b"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890527    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890530    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890533    2561 flags.go:64] FLAG: --register-node="true"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890536    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890539    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890543    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890545    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890548    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890552    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890555    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890558    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890561    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890563    2561 flags.go:64] FLAG: --runonce="false"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890566    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890569    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890572    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890575    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 12:33:32.893809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890578    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890581    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890583    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890586    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890589    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890592    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890595    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890598    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890601    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890604    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890606    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890612    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890614    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890617    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890621    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890624    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890627    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890630    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890632    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890635    2561 flags.go:64] FLAG: --v="2"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890639    2561 flags.go:64] FLAG: --version="false"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890644    2561 flags.go:64] FLAG: --vmodule=""
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890648    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.890651    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890780    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:32.894406 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890785    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890788    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890791    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890793    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890796    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890799    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890801    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890804    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890807    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890810    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890812    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890815    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890817    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890820    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890823    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890825    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890827    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890830    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890832    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890835    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:32.895034 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890837    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890840    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890842    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890845    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890847    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890850    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890852    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890855    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890857    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890862    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890864    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890867    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890869    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890872    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890874    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890877    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890879    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890882    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890884    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890887    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:32.895540 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890889    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890892    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890894    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890898    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890901    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890904    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890906    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890909    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890911    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890914    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890916    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890919    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890921    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890924    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890926    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890929    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890931    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890934    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890936    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890939    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:32.896104 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890945    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890947    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890950    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890953    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890955    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890957    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890960    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890964    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890966    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890969    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890972    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890975    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890977    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890980    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890983    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890985    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890988    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890990    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890993    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:32.896601 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890995    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.890998    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891000    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891003    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891005    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891008    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891010    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891014    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891018    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891021    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891024    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891027    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891030    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891033    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891036    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891039    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891042    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891045    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:32.897090 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.891047    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:32.897563 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.891675    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:33:32.898536 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.898515    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 12:33:32.898575 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.898536    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 12:33:32.898575 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.898558    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 12:33:32.898575 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.898567    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898622    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898628    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898631    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898635    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898638    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898640    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898643    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898646    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898648    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898651    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898653    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:32.898654 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898657    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898662    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898666    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898669    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898672    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898674    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898677    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898680    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898682    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898685    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898688    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898690    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898693    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898695    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898698    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898700    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898703    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898707    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898711    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898715    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:32.898974 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898718    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898720    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898723    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898725    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898742    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898745    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898747    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898750    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898752    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898755    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898757    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898760    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898763    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898765    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898768    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898771    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898774    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898777    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898779    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898782    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:32.899475 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898786    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898790    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898793    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898795    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898798    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898801    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898804    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898806    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898809    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898812    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898815    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898818    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898821    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898823    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898825    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898828    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898831    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898833    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898836    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:32.900050 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898838    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898841    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898843    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898846    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898848    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898851    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898853    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898857    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898860    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898863    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898865    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898868    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898870    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898872    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898875    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898878    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898880    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898883    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898885    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898888    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:32.900548 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898891    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898893    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898895    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898898    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898900    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898903    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898906    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898909    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.898911    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.898916    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899027    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899031    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899033    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899036    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899039    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:32.901049 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899041    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899044    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899047    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899050    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899052    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899055    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899058    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899061    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899064    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899066    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899069    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899071    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899074    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899077    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899081    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899084    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899087    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899090    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899093    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:32.901418 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899096    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899098    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899101    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899104    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899107    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899109    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899112    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899114    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899117    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899119    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899122    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899125    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899129    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899131    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899134    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899137    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899140    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899144    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899147    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:32.901906 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899149    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899152    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899155    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899158    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899161    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899164    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899167    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899170    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899172    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899175    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899177    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899179    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899182    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899184    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899187    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899189    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899192    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899194    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899197    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899199    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:32.902360 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899202    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899205    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899207    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899210    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899212    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899215    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899218    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899220    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899223    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899226    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899229    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899231    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899234    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899236    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899239    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899241    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899244    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899246    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899249    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899251    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:32.902855 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899254    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899256    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899259    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899261    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899264    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899266    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899269    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899271    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899274    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899276    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899278    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899282    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899284    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899287    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899289    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:32.903333 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:32.899292    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:32.903693 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.899297    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:33:32.903693 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.899417    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 12:33:32.903693 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.901396    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 12:33:32.903693 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.902868    2561 server.go:1019] "Starting client certificate rotation"
Jul 28 12:33:32.903693 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.902962    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:33:32.903693 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.902997    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:33:32.905189 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.905176    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:33:32.906356 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.906336    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:33:32.925984 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.925967    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 28 12:33:32.928345 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.928328    2561 log.go:25] "Validated CRI v1 image API"
Jul 28 12:33:32.929206 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.929190    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 12:33:32.932870 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.932851    2561 fs.go:135] Filesystem UUIDs: map[408e24c0-feef-4acd-8935-a6849f35b899:/dev/nvme0n1p3 75f7d33f-6a01-47df-98d1-c4bfcc9469e1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 28 12:33:32.932927 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.932869    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 12:33:32.939158 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.939048    2561 manager.go:216] Machine: {Timestamp:2026-07-28 12:33:32.937687661 +0000 UTC m=+0.297292257 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100374 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23997fddea586aa2df83fda75679d8 SystemUUID:ec23997f-ddea-586a-a2df-83fda75679d8 BootID:54b14eda-6bb8-41ba-aa41-c03e57fbd108 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0a:5d:33:7d:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0a:5d:33:7d:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:c2:7e:29:33:b3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 12:33:32.939158 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.939154    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 12:33:32.939274 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.939263    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 12:33:32.942535 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.942509    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 12:33:32.942667 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.942538    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-58.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 12:33:32.942930 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.942916    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:33:32.943344 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.943333    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 12:33:32.943382 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.943346    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 12:33:32.943382 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.943354    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:33:32.944785 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.944774    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:33:32.945855 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.945845    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 12:33:32.948762 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.948750    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:33:32.948875 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.948866    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 12:33:32.948906 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.948896    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 12:33:32.948906 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.948906    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 12:33:32.948964 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.948920    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 12:33:32.948964 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.948929    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 28 12:33:32.948964 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.948937    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 12:33:32.949282 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.949269    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:33:32.949314 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.949292    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:33:32.951623 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.951604    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 12:33:32.953883 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.953866    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 12:33:32.953943 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.953899    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 12:33:32.955908 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955896    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 12:33:32.955953 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955913    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 12:33:32.955953 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955921    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 12:33:32.955953 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955926    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 12:33:32.955953 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955935    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 12:33:32.955953 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955948    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 12:33:32.956087 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955957    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 12:33:32.956087 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955962    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 12:33:32.956087 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955968    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 12:33:32.956087 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955973    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 12:33:32.956087 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955985    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 12:33:32.956087 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.955990    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 12:33:32.957820 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.957779    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 12:33:32.958035 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.958022    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 12:33:32.960954 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.960933    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-58.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 12:33:32.961045 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:32.960997    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 12:33:32.961045 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:32.960997    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 12:33:32.961632 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.961620    2561 server.go:1268] "Started kubelet"
Jul 28 12:33:32.961714 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.961688    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 12:33:32.962120 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.961926    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 12:33:32.962289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.962172    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 12:33:32.962625 ip-10-0-138-58 systemd[1]: Started Kubernetes Kubelet.
Jul 28 12:33:32.964399 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.964382    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 12:33:32.966852 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.966834    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 12:33:32.967616 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.967584    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pmwtz"
Jul 28 12:33:32.969145 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:32.969124    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 12:33:32.970187 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:32.969071    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-58.ec2.internal.18c67426f4adb4a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-58.ec2.internal,UID:ip-10-0-138-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-58.ec2.internal,},FirstTimestamp:2026-07-28 12:33:32.961584291 +0000 UTC m=+0.321188896,LastTimestamp:2026-07-28 12:33:32.961584291 +0000 UTC m=+0.321188896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-58.ec2.internal,}"
Jul 28 12:33:32.975653 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.975634    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 12:33:32.975772 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.975633    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 12:33:32.975772 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:32.975713    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:32.975938 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.975918    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pmwtz"
Jul 28 12:33:32.976373 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976351    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 12:33:32.976373 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976370    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 12:33:32.976479 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976384    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 12:33:32.976479 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976459    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 12:33:32.976479 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976468    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 12:33:32.976647 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976590    2561 factory.go:55] Registering systemd factory
Jul 28 12:33:32.976647 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976607    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 28 12:33:32.976908 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976893    2561 factory.go:153] Registering CRI-O factory
Jul 28 12:33:32.976971 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976910    2561 factory.go:223] Registration of the crio container factory successfully
Jul 28 12:33:32.977039 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976972    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 12:33:32.977039 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.976995    2561 factory.go:103] Registering Raw factory
Jul 28 12:33:32.977039 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.977010    2561 manager.go:1196] Started watching for new ooms in manager
Jul 28 12:33:32.978942 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.978925    2561 manager.go:349] Starting recovery of all containers
Jul 28 12:33:32.988545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.988524    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:32.990017 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.990004    2561 manager.go:354] Recovery completed
Jul 28 12:33:32.991931 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:32.991850    2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-58.ec2.internal\" not found" node="ip-10-0-138-58.ec2.internal"
Jul 28 12:33:32.996348 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.996332    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:32.998208 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.998193    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:32.998293 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.998227    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:32.998293 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.998242    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:32.998879 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.998861    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 12:33:32.998879 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.998875    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 12:33:32.998987 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.998891    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:33:32.998987 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.998868    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:32.999693 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.999676    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:32.999783 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.999706    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:32.999783 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:32.999716    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:33.001561 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.001546    2561 policy_none.go:49] "None policy: Start"
Jul 28 12:33:33.001561 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.001563    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 12:33:33.001653 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.001573    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 12:33:33.003590 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.003574    2561 policy_none.go:47] "Start"
Jul 28 12:33:33.048384 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.048367    2561 manager.go:337] "Starting Device Plugin manager"
Jul 28 12:33:33.063525 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.048407    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 12:33:33.063525 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.048419    2561 server.go:87] "Starting device plugin registration server"
Jul 28 12:33:33.063525 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.048703    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 12:33:33.063525 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.048716    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 12:33:33.063525 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.048852    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 12:33:33.063525 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.048936    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 12:33:33.063525 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.048946    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 12:33:33.063525 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.049508    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 12:33:33.063525 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.049550    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:33.094808 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.094641    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 12:33:33.096061 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.096040    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 12:33:33.096123 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.096095    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 12:33:33.096123 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.096119    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 12:33:33.096187 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.096161    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 12:33:33.099247 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.099233    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:33.149783 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.149672    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:33.150606 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.150591    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:33.150688 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.150619    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:33.150688 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.150628    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:33.150688 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.150652    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.167171 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.167153    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.167230 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.167176    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-58.ec2.internal\": node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:33.196947 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.196917    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal"]
Jul 28 12:33:33.197011 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.196996    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:33.197850 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.197833    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:33.197910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.197865    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:33.197910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.197875    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:33.200231 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.200218    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:33.200303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.200289    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.200350 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.200321    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:33.200930 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.200916    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:33.201010 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.200944    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:33.201010 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.200957    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:33.201010 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.200967    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:33.201010 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.200985    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:33.201010 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.200993    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:33.203276 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.203263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.203356 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.203290    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:33.203922 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.203907    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:33.203992 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.203932    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:33.203992 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.203968    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:33.210240 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.210224    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:33.223318 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.223300    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-58.ec2.internal\" not found" node="ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.227364 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.227349    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-58.ec2.internal\" not found" node="ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.311110 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.311084    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:33.377268 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.377243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d8f03f8e6759f6f8b7bccae6a6de85c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-58.ec2.internal\" (UID: \"3d8f03f8e6759f6f8b7bccae6a6de85c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.377355 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.377271    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da82f851d450a2805057d5c7cb6e4836-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal\" (UID: \"da82f851d450a2805057d5c7cb6e4836\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.377355 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.377296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da82f851d450a2805057d5c7cb6e4836-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal\" (UID: \"da82f851d450a2805057d5c7cb6e4836\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.411647 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.411592    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:33.477769 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.477715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da82f851d450a2805057d5c7cb6e4836-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal\" (UID: \"da82f851d450a2805057d5c7cb6e4836\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.477769 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.477777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da82f851d450a2805057d5c7cb6e4836-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal\" (UID: \"da82f851d450a2805057d5c7cb6e4836\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.477972 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.477794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d8f03f8e6759f6f8b7bccae6a6de85c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-58.ec2.internal\" (UID: \"3d8f03f8e6759f6f8b7bccae6a6de85c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.477972 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.477833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d8f03f8e6759f6f8b7bccae6a6de85c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-58.ec2.internal\" (UID: \"3d8f03f8e6759f6f8b7bccae6a6de85c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.477972 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.477838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da82f851d450a2805057d5c7cb6e4836-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal\" (UID: \"da82f851d450a2805057d5c7cb6e4836\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.477972 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.477860    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da82f851d450a2805057d5c7cb6e4836-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal\" (UID: \"da82f851d450a2805057d5c7cb6e4836\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.511888 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.511861    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:33.526069 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.526045    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.531966 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.531131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:33.612522 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.612488    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:33.713001 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.712935    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:33.813515 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.813481    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:33.850749 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.850700    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:33.902498 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.902465    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 12:33:33.902991 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.902646    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:33:33.902991 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.902667    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:33:33.902991 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.902667    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:33:33.913772 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:33.913751    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:33.985681 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.985602    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:28:32 +0000 UTC" deadline="2028-05-12 03:59:37.03322721 +0000 UTC"
Jul 28 12:33:33.985681 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:33.985632    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15687h26m3.047599464s"
Jul 28 12:33:34.014808 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:34.014780    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:34.115171 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:34.115141    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:34.215696 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:34.215666    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:34.261055 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:34.261006    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda82f851d450a2805057d5c7cb6e4836.slice/crio-089d387a9a92e3229bdb8c5c3557b42c252034b5f7403ae1d163359a45e74353 WatchSource:0}: Error finding container 089d387a9a92e3229bdb8c5c3557b42c252034b5f7403ae1d163359a45e74353: Status 404 returned error can't find the container with id 089d387a9a92e3229bdb8c5c3557b42c252034b5f7403ae1d163359a45e74353
Jul 28 12:33:34.261677 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:34.261652    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8f03f8e6759f6f8b7bccae6a6de85c.slice/crio-a8636f8330511d0131664f5f39f99205d6c3bf7806c4c3adcfa09e3dfb788f1b WatchSource:0}: Error finding container a8636f8330511d0131664f5f39f99205d6c3bf7806c4c3adcfa09e3dfb788f1b: Status 404 returned error can't find the container with id a8636f8330511d0131664f5f39f99205d6c3bf7806c4c3adcfa09e3dfb788f1b
Jul 28 12:33:34.265616 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.265596    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:33:34.316244 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:34.316210    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:34.416922 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:34.416894    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-58.ec2.internal\" not found"
Jul 28 12:33:34.448695 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.447478    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:34.476563 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.476539    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:34.486950 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.486927    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:33:34.487802 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.487789    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal"
Jul 28 12:33:34.497516 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.497494    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:33:34.847747 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.847710    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:34.950801 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.950779    2561 apiserver.go:52] "Watching apiserver"
Jul 28 12:33:34.958637 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.958619    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 12:33:34.959001 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.958982    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-zn6gv","openshift-network-operator/iptables-alerter-xp9lg","openshift-ovn-kubernetes/ovnkube-node-qbd9x","kube-system/konnectivity-agent-kt6wr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx","openshift-cluster-node-tuning-operator/tuned-zl6cw","openshift-image-registry/node-ca-28qkt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal","openshift-multus/multus-additional-cni-plugins-d5d5j","openshift-multus/multus-m57qs","openshift-multus/network-metrics-daemon-tjcgm","kube-system/global-pull-secret-syncer-h72fp","kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal","openshift-dns/node-resolver-gwgtq"]
Jul 28 12:33:34.962759 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.962725    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:34.964749 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.964716    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:34.964833 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:34.964798    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:34.964984 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.964966    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 12:33:34.965135 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.965108    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 12:33:34.965215 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.965136    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 12:33:34.965260 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.965209    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vswcr\""
Jul 28 12:33:34.967584 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.967562    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.970371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.969958    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 12:33:34.970371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.970060    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:34.970371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.970115    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 12:33:34.970371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.970147    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 12:33:34.970371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.970155    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 12:33:34.970371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.970338    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 12:33:34.970715 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.970589    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 12:33:34.971658 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.971640    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fm9ds\""
Jul 28 12:33:34.972174 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.972161    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 12:33:34.972508 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.972488    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ft277\""
Jul 28 12:33:34.972665 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.972530    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:34.972748 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.972716    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 12:33:34.974830 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.974791    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 12:33:34.974954 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.974936    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 12:33:34.975052 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.975034    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bp6n6\""
Jul 28 12:33:34.975097 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.975074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.976319 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.976300    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 12:33:34.977266 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.977250    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 12:33:34.978443 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.978422    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:33:34.978777 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.978761    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bklq6\""
Jul 28 12:33:34.978860 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.978815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:34.979387 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.979371    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 12:33:34.980953 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.980934    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:34.981185 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.981162    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 12:33:34.981910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.981202    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 12:33:34.981910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.981397    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 12:33:34.981910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.981417    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 12:33:34.981910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.981458    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 12:33:34.981910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.981488    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-99pgh\""
Jul 28 12:33:34.983162 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.983143    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 12:33:34.983162 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.983162    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c89hh\""
Jul 28 12:33:34.983477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.983461    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:34.983644 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.983491    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 12:33:34.984529 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.984509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-lib-modules\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.984626 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.984536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc255e86-f3aa-4374-9f7a-c4de06be12a6-host\") pod \"node-ca-28qkt\" (UID: \"fc255e86-f3aa-4374-9f7a-c4de06be12a6\") " pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:34.984626 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.984553    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6jqk\" (UniqueName: \"kubernetes.io/projected/18584de2-6d71-420c-a072-9b4ff6950071-kube-api-access-r6jqk\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:34.984626 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.984569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-node-log\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.984626 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.984606    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-socket-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:34.984923 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.984629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-var-lib-kubelet\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.984969 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.984941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-host\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.984969 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.984959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcccf\" (UniqueName: \"kubernetes.io/projected/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-kube-api-access-qcccf\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.985046 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.984973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-var-lib-openvswitch\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985046 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.984989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985046 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20f422dd-5142-4477-a522-4b50a217e8b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:34.985046 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-sysctl-conf\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.985232 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-tmp\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.985232 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-run-openvswitch\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985232 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-ovnkube-script-lib\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985232 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985189    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc2806da-0277-4109-a56b-47fddeef6f1d-agent-certs\") pod \"konnectivity-agent-kt6wr\" (UID: \"bc2806da-0277-4109-a56b-47fddeef6f1d\") " pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:34.985232 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk79q\" (UniqueName: \"kubernetes.io/projected/20f422dd-5142-4477-a522-4b50a217e8b4-kube-api-access-mk79q\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:34.985462 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-sysconfig\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.985462 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-systemd\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.985462 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-systemd-units\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985462 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-run-ovn\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985462 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-cni-bin\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985462 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-kubernetes\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.985462 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20f422dd-5142-4477-a522-4b50a217e8b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:34.985462 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-sysctl-d\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-env-overrides\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc2806da-0277-4109-a56b-47fddeef6f1d-konnectivity-ca\") pod \"konnectivity-agent-kt6wr\" (UID: \"bc2806da-0277-4109-a56b-47fddeef6f1d\") " pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985633    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wcbx6\""
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-sys-fs\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-modprobe-d\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-run-systemd\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-device-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9kzf\" (UniqueName: \"kubernetes.io/projected/b26a0b4d-32de-452d-a93a-b56656c4b632-kube-api-access-z9kzf\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985791    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-kubelet\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-slash\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-run-netns\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-ovn-node-metrics-cert\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.985933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985937    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-registration-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985968    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-run\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.985997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-ovnkube-config\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986037    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l4sm\" (UniqueName: \"kubernetes.io/projected/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-kube-api-access-5l4sm\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986063    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-os-release\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986070    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-sys\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986152    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-tuned\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6n7\" (UniqueName: \"kubernetes.io/projected/fc255e86-f3aa-4374-9f7a-c4de06be12a6-kube-api-access-bx6n7\") pod \"node-ca-28qkt\" (UID: \"fc255e86-f3aa-4374-9f7a-c4de06be12a6\") " pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-log-socket\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-system-cni-dir\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986317    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m57qs"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-etc-openvswitch\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20f422dd-5142-4477-a522-4b50a217e8b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:34.986678 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-cni-netd\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:34.987480 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc255e86-f3aa-4374-9f7a-c4de06be12a6-serviceca\") pod \"node-ca-28qkt\" (UID: \"fc255e86-f3aa-4374-9f7a-c4de06be12a6\") " pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:34.987480 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-cnibin\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:34.987480 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.986466    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:33:34.988396 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.988380    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wvr8d\""
Jul 28 12:33:34.988396 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.988391    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 12:33:34.989937 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.989917    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:34.990031 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:34.989983    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:34.991271 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:34.991255    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:34.991357 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:34.991318    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:35.017883 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.017862    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4vvfl"
Jul 28 12:33:35.025984 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.025955    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4vvfl"
Jul 28 12:33:35.078943 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.078913    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 12:33:35.088322 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-daemon-config\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.088502 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088339    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-env-overrides\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.088502 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc2806da-0277-4109-a56b-47fddeef6f1d-konnectivity-ca\") pod \"konnectivity-agent-kt6wr\" (UID: \"bc2806da-0277-4109-a56b-47fddeef6f1d\") " pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:35.088502 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-sys-fs\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.088502 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-modprobe-d\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.088502 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-run-systemd\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.088502 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-run-k8s-cni-cncf-io\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.088502 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-device-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9kzf\" (UniqueName: \"kubernetes.io/projected/b26a0b4d-32de-452d-a93a-b56656c4b632-kube-api-access-z9kzf\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-kubelet\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-slash\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-run-netns\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5635b25c-2a48-4626-aeed-a2f7589b5074-cni-binary-copy\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhjl\" (UniqueName: \"kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl\") pod \"network-check-target-zn6gv\" (UID: \"04e859b2-becc-46c1-a587-64e662d550dd\") " pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-ovn-node-metrics-cert\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-socket-dir-parent\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088753    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-registration-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-run\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-ovnkube-config\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l4sm\" (UniqueName: \"kubernetes.io/projected/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-kube-api-access-5l4sm\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-os-release\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.088881 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-var-lib-cni-multus\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-var-lib-kubelet\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-hostroot\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-sys\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-tuned\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.088989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6n7\" (UniqueName: \"kubernetes.io/projected/fc255e86-f3aa-4374-9f7a-c4de06be12a6-kube-api-access-bx6n7\") pod \"node-ca-28qkt\" (UID: \"fc255e86-f3aa-4374-9f7a-c4de06be12a6\") " pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-log-socket\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-system-cni-dir\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-etc-openvswitch\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20f422dd-5142-4477-a522-4b50a217e8b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-os-release\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089204    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-cni-netd\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhjj9\" (UniqueName: \"kubernetes.io/projected/f11de13b-b398-4dc5-a41d-7e08de507c51-kube-api-access-dhjj9\") pod \"node-resolver-gwgtq\" (UID: \"f11de13b-b398-4dc5-a41d-7e08de507c51\") " pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-run-multus-certs\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089279    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-kubelet-config\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:35.089624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc255e86-f3aa-4374-9f7a-c4de06be12a6-serviceca\") pod \"node-ca-28qkt\" (UID: \"fc255e86-f3aa-4374-9f7a-c4de06be12a6\") " pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-cnibin\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zlhl\" (UniqueName: \"kubernetes.io/projected/5635b25c-2a48-4626-aeed-a2f7589b5074-kube-api-access-5zlhl\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-lib-modules\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc255e86-f3aa-4374-9f7a-c4de06be12a6-host\") pod \"node-ca-28qkt\" (UID: \"fc255e86-f3aa-4374-9f7a-c4de06be12a6\") " pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6jqk\" (UniqueName: \"kubernetes.io/projected/18584de2-6d71-420c-a072-9b4ff6950071-kube-api-access-r6jqk\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-node-log\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-dbus\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-socket-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-cni-dir\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-run-netns\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-var-lib-kubelet\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-host\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcccf\" (UniqueName: \"kubernetes.io/projected/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-kube-api-access-qcccf\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-var-lib-openvswitch\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089720    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20f422dd-5142-4477-a522-4b50a217e8b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.090532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-sysctl-conf\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-tmp\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-run-openvswitch\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-ovnkube-script-lib\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c4c911a-2c4b-4230-9e13-13f2810272c0-host-slash\") pod \"iptables-alerter-xp9lg\" (UID: \"2c4c911a-2c4b-4230-9e13-13f2810272c0\") " pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc2806da-0277-4109-a56b-47fddeef6f1d-agent-certs\") pod \"konnectivity-agent-kt6wr\" (UID: \"bc2806da-0277-4109-a56b-47fddeef6f1d\") " pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089909    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk79q\" (UniqueName: \"kubernetes.io/projected/20f422dd-5142-4477-a522-4b50a217e8b4-kube-api-access-mk79q\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f11de13b-b398-4dc5-a41d-7e08de507c51-tmp-dir\") pod \"node-resolver-gwgtq\" (UID: \"f11de13b-b398-4dc5-a41d-7e08de507c51\") " pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.089981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-cnibin\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090006    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-conf-dir\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-sysconfig\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-systemd\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-systemd-units\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090115    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-run-ovn\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-cni-bin\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-system-cni-dir\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-var-lib-cni-bin\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.091338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-etc-kubernetes\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-kubernetes\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20f422dd-5142-4477-a522-4b50a217e8b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f11de13b-b398-4dc5-a41d-7e08de507c51-hosts-file\") pod \"node-resolver-gwgtq\" (UID: \"f11de13b-b398-4dc5-a41d-7e08de507c51\") " pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090323    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbr52\" (UniqueName: \"kubernetes.io/projected/2c4c911a-2c4b-4230-9e13-13f2810272c0-kube-api-access-bbr52\") pod \"iptables-alerter-xp9lg\" (UID: \"2c4c911a-2c4b-4230-9e13-13f2810272c0\") " pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-sysctl-d\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.090440    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c4c911a-2c4b-4230-9e13-13f2810272c0-iptables-alerter-script\") pod \"iptables-alerter-xp9lg\" (UID: \"2c4c911a-2c4b-4230-9e13-13f2810272c0\") " pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.091017    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-env-overrides\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.091501    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc2806da-0277-4109-a56b-47fddeef6f1d-konnectivity-ca\") pod \"konnectivity-agent-kt6wr\" (UID: \"bc2806da-0277-4109-a56b-47fddeef6f1d\") " pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.091578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-sys-fs\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.091686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-modprobe-d\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.091746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-run-systemd\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.091804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-device-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.092079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-kubelet\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.092126    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-slash\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.092172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.092163    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-run-netns\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.092992 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.092202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.092992 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.092533    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093223    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-etc-openvswitch\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-var-lib-kubelet\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-socket-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.093240    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-cnibin\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-var-lib-openvswitch\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093316    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-registration-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.093332    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs podName:18584de2-6d71-420c-a072-9b4ff6950071 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:35.593309583 +0000 UTC m=+2.952914196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs") pod "network-metrics-daemon-tjcgm" (UID: "18584de2-6d71-420c-a072-9b4ff6950071") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093348    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-host\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-run\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-sysctl-conf\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093482    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-cni-bin\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093910    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc255e86-f3aa-4374-9f7a-c4de06be12a6-serviceca\") pod \"node-ca-28qkt\" (UID: \"fc255e86-f3aa-4374-9f7a-c4de06be12a6\") " pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.093933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20f422dd-5142-4477-a522-4b50a217e8b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-ovnkube-script-lib\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.094725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-run-openvswitch\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-kubernetes\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094254    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-node-log\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094320    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20f422dd-5142-4477-a522-4b50a217e8b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b26a0b4d-32de-452d-a93a-b56656c4b632-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-lib-modules\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094493    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-host-cni-netd\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-log-socket\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-system-cni-dir\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20f422dd-5142-4477-a522-4b50a217e8b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc255e86-f3aa-4374-9f7a-c4de06be12a6-host\") pod \"node-ca-28qkt\" (UID: \"fc255e86-f3aa-4374-9f7a-c4de06be12a6\") " pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-sys\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.094876    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-sysctl-d\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.095101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-ovnkube-config\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.095164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-os-release\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.095196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-systemd\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.095240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-sysconfig\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.095545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.095284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-systemd-units\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.096506 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.095304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20f422dd-5142-4477-a522-4b50a217e8b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.096506 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.095316    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-run-ovn\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.097824 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.097451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc2806da-0277-4109-a56b-47fddeef6f1d-agent-certs\") pod \"konnectivity-agent-kt6wr\" (UID: \"bc2806da-0277-4109-a56b-47fddeef6f1d\") " pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:35.097824 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.097771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-ovn-node-metrics-cert\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.098935 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.098773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-tmp\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.100127 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.099721    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-etc-tuned\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.100539 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.100520    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9kzf\" (UniqueName: \"kubernetes.io/projected/b26a0b4d-32de-452d-a93a-b56656c4b632-kube-api-access-z9kzf\") pod \"aws-ebs-csi-driver-node-6jpqx\" (UID: \"b26a0b4d-32de-452d-a93a-b56656c4b632\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.103316 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.103266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal" event={"ID":"da82f851d450a2805057d5c7cb6e4836","Type":"ContainerStarted","Data":"089d387a9a92e3229bdb8c5c3557b42c252034b5f7403ae1d163359a45e74353"}
Jul 28 12:33:35.104574 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.104512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6jqk\" (UniqueName: \"kubernetes.io/projected/18584de2-6d71-420c-a072-9b4ff6950071-kube-api-access-r6jqk\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:35.105168 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.105144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk79q\" (UniqueName: \"kubernetes.io/projected/20f422dd-5142-4477-a522-4b50a217e8b4-kube-api-access-mk79q\") pod \"multus-additional-cni-plugins-d5d5j\" (UID: \"20f422dd-5142-4477-a522-4b50a217e8b4\") " pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.105259 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.105203    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6n7\" (UniqueName: \"kubernetes.io/projected/fc255e86-f3aa-4374-9f7a-c4de06be12a6-kube-api-access-bx6n7\") pod \"node-ca-28qkt\" (UID: \"fc255e86-f3aa-4374-9f7a-c4de06be12a6\") " pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:35.106296 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.106270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcccf\" (UniqueName: \"kubernetes.io/projected/39f606ae-f5c0-4d7b-bd8e-f927a72245ea-kube-api-access-qcccf\") pod \"tuned-zl6cw\" (UID: \"39f606ae-f5c0-4d7b-bd8e-f927a72245ea\") " pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.107172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.107130    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l4sm\" (UniqueName: \"kubernetes.io/projected/f3e83f2c-efb6-4e20-a23c-5a24d97cb18e-kube-api-access-5l4sm\") pod \"ovnkube-node-qbd9x\" (UID: \"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.107825 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.107725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal" event={"ID":"3d8f03f8e6759f6f8b7bccae6a6de85c","Type":"ContainerStarted","Data":"a8636f8330511d0131664f5f39f99205d6c3bf7806c4c3adcfa09e3dfb788f1b"}
Jul 28 12:33:35.191619 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-cni-dir\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.191818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-run-netns\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.191818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c4c911a-2c4b-4230-9e13-13f2810272c0-host-slash\") pod \"iptables-alerter-xp9lg\" (UID: \"2c4c911a-2c4b-4230-9e13-13f2810272c0\") " pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:35.191818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f11de13b-b398-4dc5-a41d-7e08de507c51-tmp-dir\") pod \"node-resolver-gwgtq\" (UID: \"f11de13b-b398-4dc5-a41d-7e08de507c51\") " pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:35.191818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-cnibin\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.191818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-conf-dir\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.191818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-system-cni-dir\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.191818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-var-lib-cni-bin\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-etc-kubernetes\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f11de13b-b398-4dc5-a41d-7e08de507c51-hosts-file\") pod \"node-resolver-gwgtq\" (UID: \"f11de13b-b398-4dc5-a41d-7e08de507c51\") " pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbr52\" (UniqueName: \"kubernetes.io/projected/2c4c911a-2c4b-4230-9e13-13f2810272c0-kube-api-access-bbr52\") pod \"iptables-alerter-xp9lg\" (UID: \"2c4c911a-2c4b-4230-9e13-13f2810272c0\") " pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c4c911a-2c4b-4230-9e13-13f2810272c0-iptables-alerter-script\") pod \"iptables-alerter-xp9lg\" (UID: \"2c4c911a-2c4b-4230-9e13-13f2810272c0\") " pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191938    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-daemon-config\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-run-k8s-cni-cncf-io\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.191998    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5635b25c-2a48-4626-aeed-a2f7589b5074-cni-binary-copy\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhjl\" (UniqueName: \"kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl\") pod \"network-check-target-zn6gv\" (UID: \"04e859b2-becc-46c1-a587-64e662d550dd\") " pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192046    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-socket-dir-parent\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-var-lib-cni-multus\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-var-lib-kubelet\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-hostroot\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-os-release\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhjj9\" (UniqueName: \"kubernetes.io/projected/f11de13b-b398-4dc5-a41d-7e08de507c51-kube-api-access-dhjj9\") pod \"node-resolver-gwgtq\" (UID: \"f11de13b-b398-4dc5-a41d-7e08de507c51\") " pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:35.192781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-run-multus-certs\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-kubelet-config\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:35.192781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zlhl\" (UniqueName: \"kubernetes.io/projected/5635b25c-2a48-4626-aeed-a2f7589b5074-kube-api-access-5zlhl\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-dbus\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:35.192781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192491    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-dbus\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:35.192781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192554    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-cni-dir\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-run-netns\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.192781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192623    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c4c911a-2c4b-4230-9e13-13f2810272c0-host-slash\") pod \"iptables-alerter-xp9lg\" (UID: \"2c4c911a-2c4b-4230-9e13-13f2810272c0\") " pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:35.193289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-cnibin\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-conf-dir\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-socket-dir-parent\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-var-lib-cni-bin\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.192995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-var-lib-cni-multus\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.193004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f11de13b-b398-4dc5-a41d-7e08de507c51-hosts-file\") pod \"node-resolver-gwgtq\" (UID: \"f11de13b-b398-4dc5-a41d-7e08de507c51\") " pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:35.193289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.193053    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-var-lib-kubelet\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.193090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-etc-kubernetes\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.193129    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-hostroot\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.193190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-os-release\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193758 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.193440    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-run-multus-certs\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193758 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.193494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-kubelet-config\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:35.193758 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.193536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-host-run-k8s-cni-cncf-io\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.193758 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.193564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f11de13b-b398-4dc5-a41d-7e08de507c51-tmp-dir\") pod \"node-resolver-gwgtq\" (UID: \"f11de13b-b398-4dc5-a41d-7e08de507c51\") " pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:35.193758 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.193755    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c4c911a-2c4b-4230-9e13-13f2810272c0-iptables-alerter-script\") pod \"iptables-alerter-xp9lg\" (UID: \"2c4c911a-2c4b-4230-9e13-13f2810272c0\") " pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:35.193997 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.193898    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:35.193997 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.193955    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret podName:14ee1edf-eb79-49a6-b644-7fb9f2bf93d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:35.693938164 +0000 UTC m=+3.053542762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret") pod "global-pull-secret-syncer-h72fp" (UID: "14ee1edf-eb79-49a6-b644-7fb9f2bf93d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:35.194929 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.194899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5635b25c-2a48-4626-aeed-a2f7589b5074-cni-binary-copy\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.195149 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.195135    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5635b25c-2a48-4626-aeed-a2f7589b5074-system-cni-dir\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.195245 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.195019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5635b25c-2a48-4626-aeed-a2f7589b5074-multus-daemon-config\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.198864 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.198841    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:33:35.198970 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.198869    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:33:35.198970 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.198884    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qlhjl for pod openshift-network-diagnostics/network-check-target-zn6gv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:35.198970 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.198939    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl podName:04e859b2-becc-46c1-a587-64e662d550dd nodeName:}" failed. No retries permitted until 2026-07-28 12:33:35.698922564 +0000 UTC m=+3.058527160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qlhjl" (UniqueName: "kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl") pod "network-check-target-zn6gv" (UID: "04e859b2-becc-46c1-a587-64e662d550dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:35.203509 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.203480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zlhl\" (UniqueName: \"kubernetes.io/projected/5635b25c-2a48-4626-aeed-a2f7589b5074-kube-api-access-5zlhl\") pod \"multus-m57qs\" (UID: \"5635b25c-2a48-4626-aeed-a2f7589b5074\") " pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.203948 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.203923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhjj9\" (UniqueName: \"kubernetes.io/projected/f11de13b-b398-4dc5-a41d-7e08de507c51-kube-api-access-dhjj9\") pod \"node-resolver-gwgtq\" (UID: \"f11de13b-b398-4dc5-a41d-7e08de507c51\") " pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:35.204047 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.203997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbr52\" (UniqueName: \"kubernetes.io/projected/2c4c911a-2c4b-4230-9e13-13f2810272c0-kube-api-access-bbr52\") pod \"iptables-alerter-xp9lg\" (UID: \"2c4c911a-2c4b-4230-9e13-13f2810272c0\") " pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:35.274824 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.274781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-28qkt"
Jul 28 12:33:35.282637 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.282607    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:33:35.291197 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.291175    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:35.293269 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.292524    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:35.296314 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.296133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx"
Jul 28 12:33:35.300331 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:35.300299    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e83f2c_efb6_4e20_a23c_5a24d97cb18e.slice/crio-08cfe91acf5b956606092afd900812d720a39fa17194f03d01f4e5abb35f2a03 WatchSource:0}: Error finding container 08cfe91acf5b956606092afd900812d720a39fa17194f03d01f4e5abb35f2a03: Status 404 returned error can't find the container with id 08cfe91acf5b956606092afd900812d720a39fa17194f03d01f4e5abb35f2a03
Jul 28 12:33:35.303242 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.303226    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zl6cw"
Jul 28 12:33:35.307996 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:35.307971    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26a0b4d_32de_452d_a93a_b56656c4b632.slice/crio-4b03b826fef8f22f0d029dd036ff26e96b6f4d2805ae2b91e91a090968a44c33 WatchSource:0}: Error finding container 4b03b826fef8f22f0d029dd036ff26e96b6f4d2805ae2b91e91a090968a44c33: Status 404 returned error can't find the container with id 4b03b826fef8f22f0d029dd036ff26e96b6f4d2805ae2b91e91a090968a44c33
Jul 28 12:33:35.308983 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.308971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d5d5j"
Jul 28 12:33:35.315692 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.315671    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gwgtq"
Jul 28 12:33:35.316146 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:35.316123    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f606ae_f5c0_4d7b_bd8e_f927a72245ea.slice/crio-3d9833a8d9138c1e3ef17e80bdab302f4db8957db7e17d16763e5a2ab90bb781 WatchSource:0}: Error finding container 3d9833a8d9138c1e3ef17e80bdab302f4db8957db7e17d16763e5a2ab90bb781: Status 404 returned error can't find the container with id 3d9833a8d9138c1e3ef17e80bdab302f4db8957db7e17d16763e5a2ab90bb781
Jul 28 12:33:35.324204 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.324185    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xp9lg"
Jul 28 12:33:35.324451 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:35.324427    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f422dd_5142_4477_a522_4b50a217e8b4.slice/crio-e4e9c7a0a3366a4c98486590471ed95a7666239283a4ed5e76438f2109dc9106 WatchSource:0}: Error finding container e4e9c7a0a3366a4c98486590471ed95a7666239283a4ed5e76438f2109dc9106: Status 404 returned error can't find the container with id e4e9c7a0a3366a4c98486590471ed95a7666239283a4ed5e76438f2109dc9106
Jul 28 12:33:35.325995 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:35.325963    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11de13b_b398_4dc5_a41d_7e08de507c51.slice/crio-a987f441ad156d6007a78c2444f912f0cf2701b9cf064a496f64d274cdab8847 WatchSource:0}: Error finding container a987f441ad156d6007a78c2444f912f0cf2701b9cf064a496f64d274cdab8847: Status 404 returned error can't find the container with id a987f441ad156d6007a78c2444f912f0cf2701b9cf064a496f64d274cdab8847
Jul 28 12:33:35.334308 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.334286    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m57qs"
Jul 28 12:33:35.336155 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:33:35.336131    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4c911a_2c4b_4230_9e13_13f2810272c0.slice/crio-e8f89f0cb99f215daea4636f009e9387ed954b7c8d378caaaa2721c69afea81f WatchSource:0}: Error finding container e8f89f0cb99f215daea4636f009e9387ed954b7c8d378caaaa2721c69afea81f: Status 404 returned error can't find the container with id e8f89f0cb99f215daea4636f009e9387ed954b7c8d378caaaa2721c69afea81f
Jul 28 12:33:35.595810 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.595767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:35.595985 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.595933    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:35.596036 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.596008    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs podName:18584de2-6d71-420c-a072-9b4ff6950071 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:36.595987814 +0000 UTC m=+3.955592397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs") pod "network-metrics-daemon-tjcgm" (UID: "18584de2-6d71-420c-a072-9b4ff6950071") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:35.608403 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.608332    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:35.696833 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.696803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:35.696999 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.696954    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:35.697055 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.697017    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret podName:14ee1edf-eb79-49a6-b644-7fb9f2bf93d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:36.696998117 +0000 UTC m=+4.056602932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret") pod "global-pull-secret-syncer-h72fp" (UID: "14ee1edf-eb79-49a6-b644-7fb9f2bf93d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:35.797943 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:35.797907    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhjl\" (UniqueName: \"kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl\") pod \"network-check-target-zn6gv\" (UID: \"04e859b2-becc-46c1-a587-64e662d550dd\") " pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:35.798110 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.798071    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:33:35.798110 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.798094    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:33:35.798110 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.798105    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qlhjl for pod openshift-network-diagnostics/network-check-target-zn6gv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:35.798277 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:35.798159    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl podName:04e859b2-becc-46c1-a587-64e662d550dd nodeName:}" failed. No retries permitted until 2026-07-28 12:33:36.798142079 +0000 UTC m=+4.157746661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlhjl" (UniqueName: "kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl") pod "network-check-target-zn6gv" (UID: "04e859b2-becc-46c1-a587-64e662d550dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:36.028931 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.028770    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:28:34 +0000 UTC" deadline="2028-01-09 16:21:37.094151403 +0000 UTC"
Jul 28 12:33:36.028931 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.028834    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12723h48m1.065321305s"
Jul 28 12:33:36.117272 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.117230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m57qs" event={"ID":"5635b25c-2a48-4626-aeed-a2f7589b5074","Type":"ContainerStarted","Data":"50535ab80e8ea850e7ede62a2873101ce44a03ef9679170f3da0b1f21c51d779"}
Jul 28 12:33:36.118723 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.118658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xp9lg" event={"ID":"2c4c911a-2c4b-4230-9e13-13f2810272c0","Type":"ContainerStarted","Data":"e8f89f0cb99f215daea4636f009e9387ed954b7c8d378caaaa2721c69afea81f"}
Jul 28 12:33:36.122172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.122102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kt6wr" event={"ID":"bc2806da-0277-4109-a56b-47fddeef6f1d","Type":"ContainerStarted","Data":"19449fbc78dc39da113399b5a37844af7dd718d504cb6837aaa13ae82afb91cf"}
Jul 28 12:33:36.124129 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.124026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x" event={"ID":"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e","Type":"ContainerStarted","Data":"08cfe91acf5b956606092afd900812d720a39fa17194f03d01f4e5abb35f2a03"}
Jul 28 12:33:36.129199 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.129125    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gwgtq" event={"ID":"f11de13b-b398-4dc5-a41d-7e08de507c51","Type":"ContainerStarted","Data":"a987f441ad156d6007a78c2444f912f0cf2701b9cf064a496f64d274cdab8847"}
Jul 28 12:33:36.134439 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.134411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5d5j" event={"ID":"20f422dd-5142-4477-a522-4b50a217e8b4","Type":"ContainerStarted","Data":"e4e9c7a0a3366a4c98486590471ed95a7666239283a4ed5e76438f2109dc9106"}
Jul 28 12:33:36.136803 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.136776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zl6cw" event={"ID":"39f606ae-f5c0-4d7b-bd8e-f927a72245ea","Type":"ContainerStarted","Data":"3d9833a8d9138c1e3ef17e80bdab302f4db8957db7e17d16763e5a2ab90bb781"}
Jul 28 12:33:36.139161 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.139136    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx" event={"ID":"b26a0b4d-32de-452d-a93a-b56656c4b632","Type":"ContainerStarted","Data":"4b03b826fef8f22f0d029dd036ff26e96b6f4d2805ae2b91e91a090968a44c33"}
Jul 28 12:33:36.141260 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.141237    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-28qkt" event={"ID":"fc255e86-f3aa-4374-9f7a-c4de06be12a6","Type":"ContainerStarted","Data":"b320c74379c15ad1e2cbc006cbd0523652c4a1a00adc2b927a46e0d3c6a0e751"}
Jul 28 12:33:36.605247 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.605208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:36.605433 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:36.605400    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:36.605505 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:36.605466    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs podName:18584de2-6d71-420c-a072-9b4ff6950071 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:38.605445362 +0000 UTC m=+5.965049968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs") pod "network-metrics-daemon-tjcgm" (UID: "18584de2-6d71-420c-a072-9b4ff6950071") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:36.706412 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.706373    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:36.706570 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:36.706529    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:36.706644 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:36.706593    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret podName:14ee1edf-eb79-49a6-b644-7fb9f2bf93d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:38.706575115 +0000 UTC m=+6.066179703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret") pod "global-pull-secret-syncer-h72fp" (UID: "14ee1edf-eb79-49a6-b644-7fb9f2bf93d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:36.806794 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:36.806757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhjl\" (UniqueName: \"kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl\") pod \"network-check-target-zn6gv\" (UID: \"04e859b2-becc-46c1-a587-64e662d550dd\") " pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:36.806976 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:36.806961    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:33:36.807027 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:36.806981    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:33:36.807027 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:36.806994    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qlhjl for pod openshift-network-diagnostics/network-check-target-zn6gv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:36.807093 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:36.807051    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl podName:04e859b2-becc-46c1-a587-64e662d550dd nodeName:}" failed. No retries permitted until 2026-07-28 12:33:38.807034172 +0000 UTC m=+6.166638782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlhjl" (UniqueName: "kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl") pod "network-check-target-zn6gv" (UID: "04e859b2-becc-46c1-a587-64e662d550dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:37.029717 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:37.029627    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:28:34 +0000 UTC" deadline="2028-04-09 19:33:49.753581864 +0000 UTC"
Jul 28 12:33:37.029717 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:37.029673    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14911h0m12.723914087s"
Jul 28 12:33:37.096941 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:37.096911    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:37.097109 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:37.097036    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:37.097474 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:37.097452    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:37.097585 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:37.097546    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:37.097751 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:37.097712    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:37.097863 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:37.097842    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:38.621863 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:38.621821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:38.622420 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:38.622141    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:38.622420 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:38.622210    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs podName:18584de2-6d71-420c-a072-9b4ff6950071 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:42.622190957 +0000 UTC m=+9.981795543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs") pod "network-metrics-daemon-tjcgm" (UID: "18584de2-6d71-420c-a072-9b4ff6950071") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:38.723353 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:38.722700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:38.723353 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:38.722944    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:38.723353 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:38.723008    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret podName:14ee1edf-eb79-49a6-b644-7fb9f2bf93d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:42.722988868 +0000 UTC m=+10.082593467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret") pod "global-pull-secret-syncer-h72fp" (UID: "14ee1edf-eb79-49a6-b644-7fb9f2bf93d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:38.824233 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:38.823448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhjl\" (UniqueName: \"kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl\") pod \"network-check-target-zn6gv\" (UID: \"04e859b2-becc-46c1-a587-64e662d550dd\") " pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:38.824233 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:38.823653    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:33:38.824233 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:38.823673    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:33:38.824233 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:38.823686    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qlhjl for pod openshift-network-diagnostics/network-check-target-zn6gv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:38.824233 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:38.823812    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl podName:04e859b2-becc-46c1-a587-64e662d550dd nodeName:}" failed. No retries permitted until 2026-07-28 12:33:42.823727115 +0000 UTC m=+10.183331703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlhjl" (UniqueName: "kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl") pod "network-check-target-zn6gv" (UID: "04e859b2-becc-46c1-a587-64e662d550dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:39.097467 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:39.097393    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:39.097629 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:39.097522    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:39.097971 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:39.097952    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:39.098089 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:39.098043    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:39.098205 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:39.098189    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:39.098311 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:39.098290    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:41.100607 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:41.097444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:41.100607 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:41.097565    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:41.100607 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:41.097948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:41.100607 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:41.098034    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:41.100607 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:41.098083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:41.100607 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:41.098156    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:42.657098 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:42.657058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:42.657516 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:42.657209    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:42.657516 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:42.657266    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs podName:18584de2-6d71-420c-a072-9b4ff6950071 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:50.657250651 +0000 UTC m=+18.016855233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs") pod "network-metrics-daemon-tjcgm" (UID: "18584de2-6d71-420c-a072-9b4ff6950071") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:42.757667 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:42.757633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:42.757905 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:42.757815    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:42.757905 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:42.757892    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret podName:14ee1edf-eb79-49a6-b644-7fb9f2bf93d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:50.757872116 +0000 UTC m=+18.117476703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret") pod "global-pull-secret-syncer-h72fp" (UID: "14ee1edf-eb79-49a6-b644-7fb9f2bf93d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:42.858771 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:42.858663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhjl\" (UniqueName: \"kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl\") pod \"network-check-target-zn6gv\" (UID: \"04e859b2-becc-46c1-a587-64e662d550dd\") " pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:42.858953 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:42.858893    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:33:42.858953 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:42.858919    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:33:42.858953 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:42.858936    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qlhjl for pod openshift-network-diagnostics/network-check-target-zn6gv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:42.859125 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:42.858998    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl podName:04e859b2-becc-46c1-a587-64e662d550dd nodeName:}" failed. No retries permitted until 2026-07-28 12:33:50.858979209 +0000 UTC m=+18.218583807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlhjl" (UniqueName: "kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl") pod "network-check-target-zn6gv" (UID: "04e859b2-becc-46c1-a587-64e662d550dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:43.103051 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:43.103026    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:43.103159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:43.103095    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:43.103201 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:43.103187    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:43.103257 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:43.103208    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:43.103318 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:43.103287    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:43.103367 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:43.103347    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:45.100656 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:45.100302    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:45.100656 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:45.100350    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:45.100656 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:45.100423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:45.100656 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:45.100421    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:45.100656 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:45.100564    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:45.100656 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:45.100623    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:47.100179 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:47.100149    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:47.100636 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:47.100249    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:47.100636 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:47.100259    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:47.100636 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:47.100366    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:47.100636 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:47.100406    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:47.100636 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:47.100449    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:49.097257 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:49.097211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:49.097697 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:49.097211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:49.097697 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:49.097361    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:49.097697 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:49.097392    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:49.097697 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:49.097211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:49.097697 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:49.097466    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:50.715225 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:50.715174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:50.715707 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:50.715341    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:50.715707 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:50.715414    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs podName:18584de2-6d71-420c-a072-9b4ff6950071 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:06.715397517 +0000 UTC m=+34.075002117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs") pod "network-metrics-daemon-tjcgm" (UID: "18584de2-6d71-420c-a072-9b4ff6950071") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:50.816156 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:50.816113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:50.816344 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:50.816249    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:50.816344 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:50.816332    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret podName:14ee1edf-eb79-49a6-b644-7fb9f2bf93d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:06.816310606 +0000 UTC m=+34.175915203 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret") pod "global-pull-secret-syncer-h72fp" (UID: "14ee1edf-eb79-49a6-b644-7fb9f2bf93d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:50.917432 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:50.917391    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhjl\" (UniqueName: \"kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl\") pod \"network-check-target-zn6gv\" (UID: \"04e859b2-becc-46c1-a587-64e662d550dd\") " pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:50.917600 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:50.917554    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:33:50.917600 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:50.917575    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:33:50.917600 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:50.917588    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qlhjl for pod openshift-network-diagnostics/network-check-target-zn6gv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:50.917786 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:50.917655    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl podName:04e859b2-becc-46c1-a587-64e662d550dd nodeName:}" failed. No retries permitted until 2026-07-28 12:34:06.917635579 +0000 UTC m=+34.277240188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlhjl" (UniqueName: "kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl") pod "network-check-target-zn6gv" (UID: "04e859b2-becc-46c1-a587-64e662d550dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:51.096343 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:51.096312    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:51.096531 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:51.096312    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:51.096531 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:51.096454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:51.096615 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:51.096542    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:51.096615 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:51.096426    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:51.096708 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:51.096644    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:53.097903 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:53.097438    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:53.097903 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:53.097529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:53.097903 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:53.097541    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:53.097903 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:53.097650    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:53.097903 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:53.097682    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:53.097903 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:53.097773    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:55.097023 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:55.096837    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:55.097545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:55.096837    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:55.097545 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:55.097146    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:55.097545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:55.096890    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:55.097545 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:55.097178    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:55.097545 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:55.097285    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:55.195393 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:55.195362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal" event={"ID":"3d8f03f8e6759f6f8b7bccae6a6de85c","Type":"ContainerStarted","Data":"49f5a5fd34018a3bea0f193f46f6dca8facf792f19b10dae5caaecf0dc1438b1"}
Jul 28 12:33:55.197098 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:55.197072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zl6cw" event={"ID":"39f606ae-f5c0-4d7b-bd8e-f927a72245ea","Type":"ContainerStarted","Data":"10516e39d4d81ce6a4ba35ee6008c8a9c257ba854e64f991e43749b97da7f385"}
Jul 28 12:33:55.203047 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:55.203016    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m57qs" event={"ID":"5635b25c-2a48-4626-aeed-a2f7589b5074","Type":"ContainerStarted","Data":"361bbbd570fb611f2b847d1c60f7e5346b327a773e6e74dcde0133199a51be91"}
Jul 28 12:33:55.210554 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:55.210509    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-58.ec2.internal" podStartSLOduration=21.210497059 podStartE2EDuration="21.210497059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:55.209685893 +0000 UTC m=+22.569290500" watchObservedRunningTime="2026-07-28 12:33:55.210497059 +0000 UTC m=+22.570101663"
Jul 28 12:33:55.229251 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:55.229200    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m57qs" podStartSLOduration=2.828721475 podStartE2EDuration="22.229183992s" totalImagesPullingTime="19.400462517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:55.228834901 +0000 UTC m=+22.588439507" watchObservedRunningTime="2026-07-28 12:33:55.229183992 +0000 UTC m=+22.588788598"
Jul 28 12:33:55.250357 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:55.250303    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zl6cw" podStartSLOduration=2.816567193 podStartE2EDuration="22.250284439s" totalImagesPullingTime="19.433717246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:55.249708574 +0000 UTC m=+22.609313178" watchObservedRunningTime="2026-07-28 12:33:55.250284439 +0000 UTC m=+22.609889044"
Jul 28 12:33:56.206486 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.206449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-28qkt" event={"ID":"fc255e86-f3aa-4374-9f7a-c4de06be12a6","Type":"ContainerStarted","Data":"4877961b0f8adb484098bda36508554d83f762f30ad10eba99bdedc263d7c07d"}
Jul 28 12:33:56.207934 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.207907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xp9lg" event={"ID":"2c4c911a-2c4b-4230-9e13-13f2810272c0","Type":"ContainerStarted","Data":"c58acc0f21666fdf22db406afce5dd219daf6ac1ffc6417d6f748848f3b094eb"}
Jul 28 12:33:56.209211 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.209188    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kt6wr" event={"ID":"bc2806da-0277-4109-a56b-47fddeef6f1d","Type":"ContainerStarted","Data":"25cd11f63a24771dc5a208b32aeb6a1d4b311e5b8780048ada838f852a3b47ec"}
Jul 28 12:33:56.212060 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.212039    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x" event={"ID":"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e","Type":"ContainerStarted","Data":"883db17dd0036b45718ed56a26a9b63cee58c48a1ea94f5c3a87582cf50a6831"}
Jul 28 12:33:56.212125 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.212094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x" event={"ID":"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e","Type":"ContainerStarted","Data":"273805b46b0f29736204a1c3c1ca44f2ca1c2b65613eb9eafb8375b3a059ca53"}
Jul 28 12:33:56.212175 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.212129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x" event={"ID":"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e","Type":"ContainerStarted","Data":"04f890bc63144803139e0c9f3e75e78ac8bcdf7dc9812504512c8a903070a536"}
Jul 28 12:33:56.212175 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.212169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x" event={"ID":"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e","Type":"ContainerStarted","Data":"251f8e96d019a98411b3ef16c7dc65758f697b71098334fa1d6ac98daf011494"}
Jul 28 12:33:56.212254 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.212207    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x" event={"ID":"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e","Type":"ContainerStarted","Data":"d901b8417a5f895fbf70f983b8fdacb4da41986cfde0f301318ff5944edcd9f4"}
Jul 28 12:33:56.212296 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.212253    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x" event={"ID":"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e","Type":"ContainerStarted","Data":"a55084fe0459043a1de164129a22ffab2939a6ca37f64a728ad3392268d54360"}
Jul 28 12:33:56.213375 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.213355    2561 generic.go:356] "Generic (PLEG): container finished" podID="da82f851d450a2805057d5c7cb6e4836" containerID="b271c0e62f109491ae75103b33ba9259230241eeaf29e1c28b7ab4913775ef9d" exitCode=0
Jul 28 12:33:56.213458 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.213421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal" event={"ID":"da82f851d450a2805057d5c7cb6e4836","Type":"ContainerDied","Data":"b271c0e62f109491ae75103b33ba9259230241eeaf29e1c28b7ab4913775ef9d"}
Jul 28 12:33:56.214767 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.214725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gwgtq" event={"ID":"f11de13b-b398-4dc5-a41d-7e08de507c51","Type":"ContainerStarted","Data":"dc6651e0e6b91527c62095789cdfeb0d3663154109b1ce407d7d117d71a50f57"}
Jul 28 12:33:56.217090 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.216848    2561 generic.go:356] "Generic (PLEG): container finished" podID="20f422dd-5142-4477-a522-4b50a217e8b4" containerID="1b51ea370984d35079f7fade830d11289bf8b35038402af9d113d1d9889661d8" exitCode=0
Jul 28 12:33:56.217228 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.217203    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5d5j" event={"ID":"20f422dd-5142-4477-a522-4b50a217e8b4","Type":"ContainerDied","Data":"1b51ea370984d35079f7fade830d11289bf8b35038402af9d113d1d9889661d8"}
Jul 28 12:33:56.218830 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.218781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx" event={"ID":"b26a0b4d-32de-452d-a93a-b56656c4b632","Type":"ContainerStarted","Data":"dd6b0ace72a0ed1a7d295d5a55b8064a6932a1b4c6917863ec562b5131153927"}
Jul 28 12:33:56.224521 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.224479    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-28qkt" podStartSLOduration=8.248666888 podStartE2EDuration="23.224461397s" totalImagesPullingTime="14.975794509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:56.223961263 +0000 UTC m=+23.583565867" watchObservedRunningTime="2026-07-28 12:33:56.224461397 +0000 UTC m=+23.584066006"
Jul 28 12:33:56.238555 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.238520    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kt6wr" podStartSLOduration=4.240372842 podStartE2EDuration="23.238507075s" totalImagesPullingTime="18.998134233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:56.238498601 +0000 UTC m=+23.598103198" watchObservedRunningTime="2026-07-28 12:33:56.238507075 +0000 UTC m=+23.598111658"
Jul 28 12:33:56.280115 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.279936    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gwgtq" podStartSLOduration=4.093675321 podStartE2EDuration="23.279921341s" totalImagesPullingTime="19.18624602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:56.279697492 +0000 UTC m=+23.639302098" watchObservedRunningTime="2026-07-28 12:33:56.279921341 +0000 UTC m=+23.639525946"
Jul 28 12:33:56.320685 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.320622    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xp9lg" podStartSLOduration=4.112144979 podStartE2EDuration="23.320600898s" totalImagesPullingTime="19.208455919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:56.299879242 +0000 UTC m=+23.659483848" watchObservedRunningTime="2026-07-28 12:33:56.320600898 +0000 UTC m=+23.680205505"
Jul 28 12:33:56.881500 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:56.881474    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 12:33:57.048450 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.048401    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:33:57.066063 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.065953    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:33:56.881494809Z","UUID":"3e27780b-ddc1-46f5-9348-879e2dcab151","Handler":null,"Name":"","Endpoint":""}
Jul 28 12:33:57.068107 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.068084    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 12:33:57.068259 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.068124    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 12:33:57.096394 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.096357    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:57.096575 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.096357    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:57.096575 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:57.096472    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:57.096575 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:57.096552    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:57.096575 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.096357    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:57.096915 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:57.096641    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:57.142861 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.142828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:33:57.223371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.223331    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal" event={"ID":"da82f851d450a2805057d5c7cb6e4836","Type":"ContainerStarted","Data":"9000b28495d80b4db8fce6411cdaa43eb6a563fe2f8366d918263be1ea3bcf8b"}
Jul 28 12:33:57.225749 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.225673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx" event={"ID":"b26a0b4d-32de-452d-a93a-b56656c4b632","Type":"ContainerStarted","Data":"139d9b9105f25c6d610a3f578a032c1910521bcbcb7d7ddf2b0f94bbacc6e5a1"}
Jul 28 12:33:57.252140 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.252093    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-58.ec2.internal" podStartSLOduration=23.252077067 podStartE2EDuration="23.252077067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:57.252070532 +0000 UTC m=+24.611675147" watchObservedRunningTime="2026-07-28 12:33:57.252077067 +0000 UTC m=+24.611681672"
Jul 28 12:33:57.913396 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.913359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:57.914173 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:57.914150    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:58.230795 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:58.230693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx" event={"ID":"b26a0b4d-32de-452d-a93a-b56656c4b632","Type":"ContainerStarted","Data":"aaf75d908475fc6b90885d6780d1ee22de788d101a786436298296c3d2b43084"}
Jul 28 12:33:58.234469 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:58.234438    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x" event={"ID":"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e","Type":"ContainerStarted","Data":"ba63decf683e991a4f9aec5e8f0533d56bb658f3331e5d461d0cbd713157548e"}
Jul 28 12:33:58.235019 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:58.235000    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:58.235498 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:58.235478    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kt6wr"
Jul 28 12:33:58.281213 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:58.281157    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jpqx" podStartSLOduration=3.792646532 podStartE2EDuration="25.28113864s" totalImagesPullingTime="21.488492108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:58.256751419 +0000 UTC m=+25.616356026" watchObservedRunningTime="2026-07-28 12:33:58.28113864 +0000 UTC m=+25.640743245"
Jul 28 12:33:58.359302 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:58.359260    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:33:59.097168 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:59.097126    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:33:59.097168 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:59.097140    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:33:59.097425 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:59.097125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:33:59.097425 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:59.097251    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:33:59.097425 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:59.097408    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:33:59.097544 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:33:59.097491    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:33:59.430978 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:59.430880    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:33:59.504587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:33:59.504552    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:00.987943 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:00.987766    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:01.097138 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:01.097104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:34:01.097346 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:01.097212    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:34:01.097346 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:01.097104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:34:01.097346 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:01.097247    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:34:01.097346 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:01.097320    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:34:01.097611 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:01.097387    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:34:01.244496 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:01.244455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x" event={"ID":"f3e83f2c-efb6-4e20-a23c-5a24d97cb18e","Type":"ContainerStarted","Data":"43e4013935ab373a79671f37c9360d3b36eaa89260778fcf3a76bd6db9a817ec"}
Jul 28 12:34:01.244788 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:01.244774    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:34:01.246408 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:01.246384    2561 generic.go:356] "Generic (PLEG): container finished" podID="20f422dd-5142-4477-a522-4b50a217e8b4" containerID="c2c51ff71403febf7f43178ab61f6825d53b9c90fbcacde93461aec1e29533e4" exitCode=0
Jul 28 12:34:01.246530 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:01.246423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5d5j" event={"ID":"20f422dd-5142-4477-a522-4b50a217e8b4","Type":"ContainerDied","Data":"c2c51ff71403febf7f43178ab61f6825d53b9c90fbcacde93461aec1e29533e4"}
Jul 28 12:34:01.261235 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:01.261214    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:34:01.281476 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:01.281438    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x" podStartSLOduration=8.425334854 podStartE2EDuration="28.281425932s" totalImagesPullingTime="19.856091078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:01.280540887 +0000 UTC m=+28.640145491" watchObservedRunningTime="2026-07-28 12:34:01.281425932 +0000 UTC m=+28.641030537"
Jul 28 12:34:02.159183 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.159094    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:34:02.250564 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.250532    2561 generic.go:356] "Generic (PLEG): container finished" podID="20f422dd-5142-4477-a522-4b50a217e8b4" containerID="d8b396ca993becb12dba2358f791c065e9097a579a5f14c7fb2869190ac84756" exitCode=0
Jul 28 12:34:02.250792 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.250624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5d5j" event={"ID":"20f422dd-5142-4477-a522-4b50a217e8b4","Type":"ContainerDied","Data":"d8b396ca993becb12dba2358f791c065e9097a579a5f14c7fb2869190ac84756"}
Jul 28 12:34:02.251073 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.251054    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:34:02.267486 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.267463    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:34:02.602023 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.601990    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tjcgm"]
Jul 28 12:34:02.602248 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.602146    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:34:02.602319 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:02.602271    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:34:02.605933 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.605907    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zn6gv"]
Jul 28 12:34:02.606039 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.606012    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:34:02.606124 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:02.606102    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:34:02.618319 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.618292    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h72fp"]
Jul 28 12:34:02.618423 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:02.618388    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:34:02.618501 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:02.618481    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:34:04.097143 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:04.097096    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:34:04.097143 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:04.097143    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:34:04.097632 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:04.097203    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:34:04.097632 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:04.097215    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:34:04.097632 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:04.097297    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:34:04.097632 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:04.097385    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:34:04.258429 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:04.258396    2561 generic.go:356] "Generic (PLEG): container finished" podID="20f422dd-5142-4477-a522-4b50a217e8b4" containerID="da8dd3910ddcf53092b1ccc6089cede35aea4117deb56904646bfbc3580f4d67" exitCode=0
Jul 28 12:34:04.258613 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:04.258474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5d5j" event={"ID":"20f422dd-5142-4477-a522-4b50a217e8b4","Type":"ContainerDied","Data":"da8dd3910ddcf53092b1ccc6089cede35aea4117deb56904646bfbc3580f4d67"}
Jul 28 12:34:04.730183 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:04.730151    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:05.039273 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:05.039239    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:05.651279 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:05.651246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:05.713468 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:05.713432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:06.097231 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:06.097195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:34:06.097231 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:06.097216    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:34:06.097231 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:06.097238    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:34:06.097518 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.097327    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjcgm" podUID="18584de2-6d71-420c-a072-9b4ff6950071"
Jul 28 12:34:06.097518 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.097458    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn6gv" podUID="04e859b2-becc-46c1-a587-64e662d550dd"
Jul 28 12:34:06.097605 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.097540    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72fp" podUID="14ee1edf-eb79-49a6-b644-7fb9f2bf93d0"
Jul 28 12:34:06.737495 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:06.737459    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:34:06.737967 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.737618    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:06.737967 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.737689    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs podName:18584de2-6d71-420c-a072-9b4ff6950071 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:38.737670997 +0000 UTC m=+66.097275593 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs") pod "network-metrics-daemon-tjcgm" (UID: "18584de2-6d71-420c-a072-9b4ff6950071") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:06.838549 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:06.838511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:34:06.838695 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.838662    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:06.838779 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.838746    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret podName:14ee1edf-eb79-49a6-b644-7fb9f2bf93d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:38.838713556 +0000 UTC m=+66.198318139 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret") pod "global-pull-secret-syncer-h72fp" (UID: "14ee1edf-eb79-49a6-b644-7fb9f2bf93d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:06.939025 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:06.938986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhjl\" (UniqueName: \"kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl\") pod \"network-check-target-zn6gv\" (UID: \"04e859b2-becc-46c1-a587-64e662d550dd\") " pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:34:06.939224 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.939145    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:06.939224 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.939169    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:06.939224 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.939181    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qlhjl for pod openshift-network-diagnostics/network-check-target-zn6gv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:06.939389 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:06.939251    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl podName:04e859b2-becc-46c1-a587-64e662d550dd nodeName:}" failed. No retries permitted until 2026-07-28 12:34:38.939230987 +0000 UTC m=+66.298835579 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlhjl" (UniqueName: "kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl") pod "network-check-target-zn6gv" (UID: "04e859b2-becc-46c1-a587-64e662d550dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:08.036110 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.036082    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-58.ec2.internal" event="NodeReady"
Jul 28 12:34:08.036548 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.036246    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 12:34:08.096762 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.096662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:34:08.096762 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.096680    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:34:08.096762 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.096697    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:34:08.104406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.104383    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dmbdk"]
Jul 28 12:34:08.104951 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.104817    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:34:08.105045 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.105012    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7n8fl\""
Jul 28 12:34:08.105086 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.105057    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:34:08.105355 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.105338    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:34:08.105573 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.105558    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:34:08.106179 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.105668    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jpxcx\""
Jul 28 12:34:08.134386 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.134365    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6zc4d"]
Jul 28 12:34:08.134513 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.134486    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.137236 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.137219    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g8tcp\""
Jul 28 12:34:08.137561 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.137538    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 12:34:08.137667 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.137582    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 12:34:08.158315 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.158287    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dmbdk"]
Jul 28 12:34:08.158459 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.158338    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6zc4d"]
Jul 28 12:34:08.158459 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.158417    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6zc4d"
Jul 28 12:34:08.161962 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.161657    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 12:34:08.161962 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.161799    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w8fbz\""
Jul 28 12:34:08.162270 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.162252    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 12:34:08.162368 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.162290    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 12:34:08.248004 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.247970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6494845-81de-42c7-b8de-2c5419e0ba31-cert\") pod \"ingress-canary-6zc4d\" (UID: \"b6494845-81de-42c7-b8de-2c5419e0ba31\") " pod="openshift-ingress-canary/ingress-canary-6zc4d"
Jul 28 12:34:08.248193 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.248090    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9j2\" (UniqueName: \"kubernetes.io/projected/b6494845-81de-42c7-b8de-2c5419e0ba31-kube-api-access-ns9j2\") pod \"ingress-canary-6zc4d\" (UID: \"b6494845-81de-42c7-b8de-2c5419e0ba31\") " pod="openshift-ingress-canary/ingress-canary-6zc4d"
Jul 28 12:34:08.248193 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.248115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2j2\" (UniqueName: \"kubernetes.io/projected/84a04ae8-3976-4608-92b8-26002d12fb16-kube-api-access-4t2j2\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.248193 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.248161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84a04ae8-3976-4608-92b8-26002d12fb16-config-volume\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.248193 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.248192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.248423 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.248250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a04ae8-3976-4608-92b8-26002d12fb16-tmp-dir\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.349308 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.349218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84a04ae8-3976-4608-92b8-26002d12fb16-config-volume\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.349308 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.349280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.349480 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.349330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a04ae8-3976-4608-92b8-26002d12fb16-tmp-dir\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.349480 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.349351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6494845-81de-42c7-b8de-2c5419e0ba31-cert\") pod \"ingress-canary-6zc4d\" (UID: \"b6494845-81de-42c7-b8de-2c5419e0ba31\") " pod="openshift-ingress-canary/ingress-canary-6zc4d"
Jul 28 12:34:08.349480 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.349397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9j2\" (UniqueName: \"kubernetes.io/projected/b6494845-81de-42c7-b8de-2c5419e0ba31-kube-api-access-ns9j2\") pod \"ingress-canary-6zc4d\" (UID: \"b6494845-81de-42c7-b8de-2c5419e0ba31\") " pod="openshift-ingress-canary/ingress-canary-6zc4d"
Jul 28 12:34:08.349480 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.349412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2j2\" (UniqueName: \"kubernetes.io/projected/84a04ae8-3976-4608-92b8-26002d12fb16-kube-api-access-4t2j2\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.349779 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:08.349764    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:34:08.349824 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.349777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84a04ae8-3976-4608-92b8-26002d12fb16-config-volume\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.349824 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:08.349821    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls podName:84a04ae8-3976-4608-92b8-26002d12fb16 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:08.849807431 +0000 UTC m=+36.209412014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls") pod "dns-default-dmbdk" (UID: "84a04ae8-3976-4608-92b8-26002d12fb16") : secret "dns-default-metrics-tls" not found
Jul 28 12:34:08.350239 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.350222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a04ae8-3976-4608-92b8-26002d12fb16-tmp-dir\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.354460 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.354317    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6494845-81de-42c7-b8de-2c5419e0ba31-cert\") pod \"ingress-canary-6zc4d\" (UID: \"b6494845-81de-42c7-b8de-2c5419e0ba31\") " pod="openshift-ingress-canary/ingress-canary-6zc4d"
Jul 28 12:34:08.359321 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.359300    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9j2\" (UniqueName: \"kubernetes.io/projected/b6494845-81de-42c7-b8de-2c5419e0ba31-kube-api-access-ns9j2\") pod \"ingress-canary-6zc4d\" (UID: \"b6494845-81de-42c7-b8de-2c5419e0ba31\") " pod="openshift-ingress-canary/ingress-canary-6zc4d"
Jul 28 12:34:08.359457 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.359391    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2j2\" (UniqueName: \"kubernetes.io/projected/84a04ae8-3976-4608-92b8-26002d12fb16-kube-api-access-4t2j2\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.404677 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.404650    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:08.468446 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.468413    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6zc4d"
Jul 28 12:34:08.542838 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.542812    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:08.644449 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.644382    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6zc4d"]
Jul 28 12:34:08.666615 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:08.666585    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6494845_81de_42c7_b8de_2c5419e0ba31.slice/crio-61f6c9857d026428e386e577cdf7fb02904daaebe803272278ae0be7f2d328ef WatchSource:0}: Error finding container 61f6c9857d026428e386e577cdf7fb02904daaebe803272278ae0be7f2d328ef: Status 404 returned error can't find the container with id 61f6c9857d026428e386e577cdf7fb02904daaebe803272278ae0be7f2d328ef
Jul 28 12:34:08.853104 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:08.853071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:08.853230 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:08.853184    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:34:08.853298 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:08.853244    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls podName:84a04ae8-3976-4608-92b8-26002d12fb16 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:09.853229181 +0000 UTC m=+37.212833764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls") pod "dns-default-dmbdk" (UID: "84a04ae8-3976-4608-92b8-26002d12fb16") : secret "dns-default-metrics-tls" not found
Jul 28 12:34:09.270744 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.270697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6zc4d" event={"ID":"b6494845-81de-42c7-b8de-2c5419e0ba31","Type":"ContainerStarted","Data":"61f6c9857d026428e386e577cdf7fb02904daaebe803272278ae0be7f2d328ef"}
Jul 28 12:34:09.273568 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.273539    2561 generic.go:356] "Generic (PLEG): container finished" podID="20f422dd-5142-4477-a522-4b50a217e8b4" containerID="fdc5ad0826639cddb8e95bfe5cdec16a043e6a65a91f37c3639d2c6f3478b67d" exitCode=0
Jul 28 12:34:09.273691 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.273585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5d5j" event={"ID":"20f422dd-5142-4477-a522-4b50a217e8b4","Type":"ContainerDied","Data":"fdc5ad0826639cddb8e95bfe5cdec16a043e6a65a91f37c3639d2c6f3478b67d"}
Jul 28 12:34:09.295239 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.295206    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b7g7l"]
Jul 28 12:34:09.310895 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.310867    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b7g7l"]
Jul 28 12:34:09.311040 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.311005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.314420 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.313351    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 12:34:09.314420 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.313585    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 12:34:09.314420 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.313866    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nnkkf\""
Jul 28 12:34:09.314420 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.314057    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 12:34:09.314420 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.314242    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 12:34:09.333520 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.333410    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:09.356606 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.356575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.356757 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.356614    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.356898 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.356859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ff797e-a55b-4020-8e90-97d5bb21db66-data-volume\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.357033 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.356915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6lbg\" (UniqueName: \"kubernetes.io/projected/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-api-access-w6lbg\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.357033 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.356985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54ff797e-a55b-4020-8e90-97d5bb21db66-crio-socket\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.406844 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.406813    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:09.457879 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.457853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ff797e-a55b-4020-8e90-97d5bb21db66-data-volume\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.457879 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.457886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6lbg\" (UniqueName: \"kubernetes.io/projected/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-api-access-w6lbg\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.458068 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.458038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54ff797e-a55b-4020-8e90-97d5bb21db66-crio-socket\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.458130 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.458090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.458130 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.458111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.458286 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:09.458255    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:34:09.458404 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.458291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54ff797e-a55b-4020-8e90-97d5bb21db66-crio-socket\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.458404 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.458271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ff797e-a55b-4020-8e90-97d5bb21db66-data-volume\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.458404 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:09.458261    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:09.458404 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:09.458351    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm podName:54ff797e-a55b-4020-8e90-97d5bb21db66 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:09.958329735 +0000 UTC m=+37.317934340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7g7l" (UID: "54ff797e-a55b-4020-8e90-97d5bb21db66") : configmap "kube-rbac-proxy" not found
Jul 28 12:34:09.458404 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:09.458372    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls podName:54ff797e-a55b-4020-8e90-97d5bb21db66 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:09.958360503 +0000 UTC m=+37.317965085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7g7l" (UID: "54ff797e-a55b-4020-8e90-97d5bb21db66") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:09.468031 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.468000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6lbg\" (UniqueName: \"kubernetes.io/projected/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-api-access-w6lbg\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.863025 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.862984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:09.863200 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:09.863138    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:34:09.863242 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:09.863210    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls podName:84a04ae8-3976-4608-92b8-26002d12fb16 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:11.863195035 +0000 UTC m=+39.222799618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls") pod "dns-default-dmbdk" (UID: "84a04ae8-3976-4608-92b8-26002d12fb16") : secret "dns-default-metrics-tls" not found
Jul 28 12:34:09.963568 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.963530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.963790 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:09.963575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:09.963790 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:09.963671    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:34:09.963790 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:09.963717    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:09.963790 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:09.963767    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm podName:54ff797e-a55b-4020-8e90-97d5bb21db66 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:10.963745767 +0000 UTC m=+38.323350375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7g7l" (UID: "54ff797e-a55b-4020-8e90-97d5bb21db66") : configmap "kube-rbac-proxy" not found
Jul 28 12:34:09.963790 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:09.963792    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls podName:54ff797e-a55b-4020-8e90-97d5bb21db66 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:10.963776564 +0000 UTC m=+38.323381165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7g7l" (UID: "54ff797e-a55b-4020-8e90-97d5bb21db66") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:10.280615 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:10.280584    2561 generic.go:356] "Generic (PLEG): container finished" podID="20f422dd-5142-4477-a522-4b50a217e8b4" containerID="562a9646c14534320c5aa4ab92a38ae679dfa4e7e1c72fdf1527955d37990111" exitCode=0
Jul 28 12:34:10.281059 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:10.280648    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5d5j" event={"ID":"20f422dd-5142-4477-a522-4b50a217e8b4","Type":"ContainerDied","Data":"562a9646c14534320c5aa4ab92a38ae679dfa4e7e1c72fdf1527955d37990111"}
Jul 28 12:34:10.884954 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:10.884771    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gwgtq_f11de13b-b398-4dc5-a41d-7e08de507c51/dns-node-resolver/0.log"
Jul 28 12:34:10.971637 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:10.971592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:10.971844 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:10.971646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:10.971844 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:10.971752    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:34:10.971844 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:10.971816    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:10.971844 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:10.971842    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm podName:54ff797e-a55b-4020-8e90-97d5bb21db66 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:12.971819372 +0000 UTC m=+40.331423981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7g7l" (UID: "54ff797e-a55b-4020-8e90-97d5bb21db66") : configmap "kube-rbac-proxy" not found
Jul 28 12:34:10.972069 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:10.971870    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls podName:54ff797e-a55b-4020-8e90-97d5bb21db66 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:12.971858793 +0000 UTC m=+40.331463376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7g7l" (UID: "54ff797e-a55b-4020-8e90-97d5bb21db66") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:11.286158 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:11.286123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5d5j" event={"ID":"20f422dd-5142-4477-a522-4b50a217e8b4","Type":"ContainerStarted","Data":"de6c4ffb9edc902bf87dc16e11a7795a08e9e01919950a9ead4f0a58c380d379"}
Jul 28 12:34:11.314386 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:11.314327    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d5d5j" podStartSLOduration=8.295822021 podStartE2EDuration="38.314311009s" totalImagesPullingTime="30.018488988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:11.312994063 +0000 UTC m=+38.672598713" watchObservedRunningTime="2026-07-28 12:34:11.314311009 +0000 UTC m=+38.673915614"
Jul 28 12:34:11.441777 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:11.441747    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:11.485053 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:11.485025    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-28qkt_fc255e86-f3aa-4374-9f7a-c4de06be12a6/node-ca/0.log"
Jul 28 12:34:11.509747 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:11.509704    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:11.829690 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:11.829658    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:11.879956 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:11.879925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:11.880080 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:11.880066    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:34:11.880131 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:11.880124    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls podName:84a04ae8-3976-4608-92b8-26002d12fb16 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:15.880107931 +0000 UTC m=+43.239712514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls") pod "dns-default-dmbdk" (UID: "84a04ae8-3976-4608-92b8-26002d12fb16") : secret "dns-default-metrics-tls" not found
Jul 28 12:34:12.289725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:12.289692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6zc4d" event={"ID":"b6494845-81de-42c7-b8de-2c5419e0ba31","Type":"ContainerStarted","Data":"c340cb989ce29480b338c89435bd8f047faad257de2435c94df35aaf36fedd97"}
Jul 28 12:34:12.308557 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:12.308512    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6zc4d" podStartSLOduration=1.5788932070000001 podStartE2EDuration="4.308497003s" totalImagesPullingTime="2.729603796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:12.308153401 +0000 UTC m=+39.667758020" watchObservedRunningTime="2026-07-28 12:34:12.308497003 +0000 UTC m=+39.668101607"
Jul 28 12:34:12.987856 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:12.987818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:12.987856 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:12.987855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:12.988049 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:12.987963    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:12.988049 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:12.988025    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls podName:54ff797e-a55b-4020-8e90-97d5bb21db66 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:16.987999265 +0000 UTC m=+44.347603849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7g7l" (UID: "54ff797e-a55b-4020-8e90-97d5bb21db66") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:12.988389 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:12.988368    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54ff797e-a55b-4020-8e90-97d5bb21db66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:13.090696 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:13.090661    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:14.468053 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.468018    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"]
Jul 28 12:34:14.496563 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.496528    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:14.506930 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.506907    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"]
Jul 28 12:34:14.507038 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.506995    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.511316 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.511291    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 12:34:14.511635 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.511605    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 12:34:14.512033 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.511679    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 12:34:14.512033 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.511834    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sll7w\""
Jul 28 12:34:14.512033 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.511915    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:34:14.598209 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.598173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e19d70c4-98f0-4e28-9ced-f733a41d1dc4-serving-cert\") pod \"service-ca-operator-5d6c75f84f-p7sz6\" (UID: \"e19d70c4-98f0-4e28-9ced-f733a41d1dc4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.598371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.598239    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6kl\" (UniqueName: \"kubernetes.io/projected/e19d70c4-98f0-4e28-9ced-f733a41d1dc4-kube-api-access-fj6kl\") pod \"service-ca-operator-5d6c75f84f-p7sz6\" (UID: \"e19d70c4-98f0-4e28-9ced-f733a41d1dc4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.598371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.598304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e19d70c4-98f0-4e28-9ced-f733a41d1dc4-config\") pod \"service-ca-operator-5d6c75f84f-p7sz6\" (UID: \"e19d70c4-98f0-4e28-9ced-f733a41d1dc4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.605219 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.605184    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"]
Jul 28 12:34:14.629380 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.629354    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp"]
Jul 28 12:34:14.629499 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.629480    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.636170 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.636140    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 12:34:14.636284 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.636234    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 12:34:14.636750 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.636716    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nmcsd\""
Jul 28 12:34:14.636909 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.636886    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 12:34:14.637384 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.637357    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:34:14.649328 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.649306    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-jhd5j"]
Jul 28 12:34:14.649456 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.649440    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp"
Jul 28 12:34:14.660371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.660351    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dwjch\""
Jul 28 12:34:14.673518 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.673503    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.673518 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.673515    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"]
Jul 28 12:34:14.675789 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.675770    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 12:34:14.676980 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.676960    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:34:14.678406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.678387    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 12:34:14.678581 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.678445    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:34:14.678667 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.678474    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 12:34:14.678667 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.678475    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 12:34:14.685522 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.685505    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 12:34:14.686871 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.686859    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-48kc9\""
Jul 28 12:34:14.691789 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.691770    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp"]
Jul 28 12:34:14.691883 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.691799    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"]
Jul 28 12:34:14.691883 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.691818    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"]
Jul 28 12:34:14.691883 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.691830    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-jhd5j"]
Jul 28 12:34:14.691883 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.691872    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:14.696155 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.696136    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:34:14.696247 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.696198    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 12:34:14.696426 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.696403    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 12:34:14.696802 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.696788    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gjwfp\""
Jul 28 12:34:14.698680 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.698664    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56dbca26-1e1f-47ec-aa9c-88881434fa9e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf\" (UID: \"56dbca26-1e1f-47ec-aa9c-88881434fa9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.698773 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.698693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e19d70c4-98f0-4e28-9ced-f733a41d1dc4-config\") pod \"service-ca-operator-5d6c75f84f-p7sz6\" (UID: \"e19d70c4-98f0-4e28-9ced-f733a41d1dc4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.698773 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.698746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e19d70c4-98f0-4e28-9ced-f733a41d1dc4-serving-cert\") pod \"service-ca-operator-5d6c75f84f-p7sz6\" (UID: \"e19d70c4-98f0-4e28-9ced-f733a41d1dc4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.698852 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.698810    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2xpn\" (UniqueName: \"kubernetes.io/projected/56dbca26-1e1f-47ec-aa9c-88881434fa9e-kube-api-access-n2xpn\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf\" (UID: \"56dbca26-1e1f-47ec-aa9c-88881434fa9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.698889 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.698858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56dbca26-1e1f-47ec-aa9c-88881434fa9e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf\" (UID: \"56dbca26-1e1f-47ec-aa9c-88881434fa9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.698889 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.698879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtp42\" (UniqueName: \"kubernetes.io/projected/629f00bc-22c6-439d-9439-6113ad8ccef4-kube-api-access-rtp42\") pod \"volume-data-source-validator-549c6d54cf-bvwkp\" (UID: \"629f00bc-22c6-439d-9439-6113ad8ccef4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp"
Jul 28 12:34:14.698958 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.698922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6kl\" (UniqueName: \"kubernetes.io/projected/e19d70c4-98f0-4e28-9ced-f733a41d1dc4-kube-api-access-fj6kl\") pod \"service-ca-operator-5d6c75f84f-p7sz6\" (UID: \"e19d70c4-98f0-4e28-9ced-f733a41d1dc4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.699312 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.699295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e19d70c4-98f0-4e28-9ced-f733a41d1dc4-config\") pod \"service-ca-operator-5d6c75f84f-p7sz6\" (UID: \"e19d70c4-98f0-4e28-9ced-f733a41d1dc4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.702365 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.702349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e19d70c4-98f0-4e28-9ced-f733a41d1dc4-serving-cert\") pod \"service-ca-operator-5d6c75f84f-p7sz6\" (UID: \"e19d70c4-98f0-4e28-9ced-f733a41d1dc4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.709938 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.709919    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:14.713766 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.713719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6kl\" (UniqueName: \"kubernetes.io/projected/e19d70c4-98f0-4e28-9ced-f733a41d1dc4-kube-api-access-fj6kl\") pod \"service-ca-operator-5d6c75f84f-p7sz6\" (UID: \"e19d70c4-98f0-4e28-9ced-f733a41d1dc4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.799808 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.799775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39287a4b-97a2-495a-a4aa-e284ef517bf6-trusted-ca\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.799969 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.799833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66q5b\" (UniqueName: \"kubernetes.io/projected/39287a4b-97a2-495a-a4aa-e284ef517bf6-kube-api-access-66q5b\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.799969 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.799913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56dbca26-1e1f-47ec-aa9c-88881434fa9e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf\" (UID: \"56dbca26-1e1f-47ec-aa9c-88881434fa9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.799969 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.799942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39287a4b-97a2-495a-a4aa-e284ef517bf6-config\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.800071 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.799974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39287a4b-97a2-495a-a4aa-e284ef517bf6-serving-cert\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.800071 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.800024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2xpn\" (UniqueName: \"kubernetes.io/projected/56dbca26-1e1f-47ec-aa9c-88881434fa9e-kube-api-access-n2xpn\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf\" (UID: \"56dbca26-1e1f-47ec-aa9c-88881434fa9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.800071 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.800041    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshvc\" (UniqueName: \"kubernetes.io/projected/b70dd332-8703-4163-a5f5-7aa2d722143b-kube-api-access-vshvc\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:14.800071 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.800066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56dbca26-1e1f-47ec-aa9c-88881434fa9e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf\" (UID: \"56dbca26-1e1f-47ec-aa9c-88881434fa9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.800193 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.800085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtp42\" (UniqueName: \"kubernetes.io/projected/629f00bc-22c6-439d-9439-6113ad8ccef4-kube-api-access-rtp42\") pod \"volume-data-source-validator-549c6d54cf-bvwkp\" (UID: \"629f00bc-22c6-439d-9439-6113ad8ccef4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp"
Jul 28 12:34:14.800193 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.800110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:14.800526 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.800510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56dbca26-1e1f-47ec-aa9c-88881434fa9e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf\" (UID: \"56dbca26-1e1f-47ec-aa9c-88881434fa9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.802867 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.802847    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:14.808875 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.808857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2xpn\" (UniqueName: \"kubernetes.io/projected/56dbca26-1e1f-47ec-aa9c-88881434fa9e-kube-api-access-n2xpn\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf\" (UID: \"56dbca26-1e1f-47ec-aa9c-88881434fa9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.809288 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.809269    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtp42\" (UniqueName: \"kubernetes.io/projected/629f00bc-22c6-439d-9439-6113ad8ccef4-kube-api-access-rtp42\") pod \"volume-data-source-validator-549c6d54cf-bvwkp\" (UID: \"629f00bc-22c6-439d-9439-6113ad8ccef4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp"
Jul 28 12:34:14.816326 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.816308    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56dbca26-1e1f-47ec-aa9c-88881434fa9e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf\" (UID: \"56dbca26-1e1f-47ec-aa9c-88881434fa9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.818361 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.818345    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"
Jul 28 12:34:14.901020 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.900991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39287a4b-97a2-495a-a4aa-e284ef517bf6-serving-cert\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.901162 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.901008    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:14.901162 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.901052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshvc\" (UniqueName: \"kubernetes.io/projected/b70dd332-8703-4163-a5f5-7aa2d722143b-kube-api-access-vshvc\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:14.901162 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.901092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:14.901162 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.901118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39287a4b-97a2-495a-a4aa-e284ef517bf6-trusted-ca\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.901371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.901175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66q5b\" (UniqueName: \"kubernetes.io/projected/39287a4b-97a2-495a-a4aa-e284ef517bf6-kube-api-access-66q5b\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.901371 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.901221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39287a4b-97a2-495a-a4aa-e284ef517bf6-config\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.902189 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:14.902023    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:34:14.902189 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.902048    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39287a4b-97a2-495a-a4aa-e284ef517bf6-config\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.902189 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:14.902111    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls podName:b70dd332-8703-4163-a5f5-7aa2d722143b nodeName:}" failed. No retries permitted until 2026-07-28 12:34:15.402094353 +0000 UTC m=+42.761698951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls") pod "cluster-samples-operator-6748489656-jt876" (UID: "b70dd332-8703-4163-a5f5-7aa2d722143b") : secret "samples-operator-tls" not found
Jul 28 12:34:14.902879 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.902857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39287a4b-97a2-495a-a4aa-e284ef517bf6-trusted-ca\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.904055 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.904037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39287a4b-97a2-495a-a4aa-e284ef517bf6-serving-cert\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.912622 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.912591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66q5b\" (UniqueName: \"kubernetes.io/projected/39287a4b-97a2-495a-a4aa-e284ef517bf6-kube-api-access-66q5b\") pod \"console-operator-584dcb6b54-jhd5j\" (UID: \"39287a4b-97a2-495a-a4aa-e284ef517bf6\") " pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.913264 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.913245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshvc\" (UniqueName: \"kubernetes.io/projected/b70dd332-8703-4163-a5f5-7aa2d722143b-kube-api-access-vshvc\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:14.938333 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.938305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"
Jul 28 12:34:14.957630 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.957595    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp"
Jul 28 12:34:14.984691 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.982504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:14.984691 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.983971    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6"]
Jul 28 12:34:14.987310 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:14.987282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:14.989218 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:14.989178    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19d70c4_98f0_4e28_9ced_f733a41d1dc4.slice/crio-8e5caa5923a588be4cc090e4d3b554bc26e651ba7aafd97af8a282e089297769 WatchSource:0}: Error finding container 8e5caa5923a588be4cc090e4d3b554bc26e651ba7aafd97af8a282e089297769: Status 404 returned error can't find the container with id 8e5caa5923a588be4cc090e4d3b554bc26e651ba7aafd97af8a282e089297769
Jul 28 12:34:15.121591 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.121558    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf"]
Jul 28 12:34:15.125726 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:15.125696    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dbca26_1e1f_47ec_aa9c_88881434fa9e.slice/crio-18b8ee99010954834dba2c49f78a97c28fa0b1cdc3ea10900138f4d02e21c610 WatchSource:0}: Error finding container 18b8ee99010954834dba2c49f78a97c28fa0b1cdc3ea10900138f4d02e21c610: Status 404 returned error can't find the container with id 18b8ee99010954834dba2c49f78a97c28fa0b1cdc3ea10900138f4d02e21c610
Jul 28 12:34:15.144458 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.144425    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp"]
Jul 28 12:34:15.148017 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:15.147992    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629f00bc_22c6_439d_9439_6113ad8ccef4.slice/crio-c56922778319242c9c43c166ddffd61c34da0e425e45ceae0deeebd2c0819ce3 WatchSource:0}: Error finding container c56922778319242c9c43c166ddffd61c34da0e425e45ceae0deeebd2c0819ce3: Status 404 returned error can't find the container with id c56922778319242c9c43c166ddffd61c34da0e425e45ceae0deeebd2c0819ce3
Jul 28 12:34:15.187974 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.187942    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-jhd5j"]
Jul 28 12:34:15.191101 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:15.191076    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39287a4b_97a2_495a_a4aa_e284ef517bf6.slice/crio-a0429ec32aa8ecc0a21f9c3b16a8f107838c7b1bc351ac92cc332d5d891bad6d WatchSource:0}: Error finding container a0429ec32aa8ecc0a21f9c3b16a8f107838c7b1bc351ac92cc332d5d891bad6d: Status 404 returned error can't find the container with id a0429ec32aa8ecc0a21f9c3b16a8f107838c7b1bc351ac92cc332d5d891bad6d
Jul 28 12:34:15.296923 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.296887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf" event={"ID":"56dbca26-1e1f-47ec-aa9c-88881434fa9e","Type":"ContainerStarted","Data":"18b8ee99010954834dba2c49f78a97c28fa0b1cdc3ea10900138f4d02e21c610"}
Jul 28 12:34:15.298032 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.298002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6" event={"ID":"e19d70c4-98f0-4e28-9ced-f733a41d1dc4","Type":"ContainerStarted","Data":"8e5caa5923a588be4cc090e4d3b554bc26e651ba7aafd97af8a282e089297769"}
Jul 28 12:34:15.299071 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.299036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" event={"ID":"39287a4b-97a2-495a-a4aa-e284ef517bf6","Type":"ContainerStarted","Data":"a0429ec32aa8ecc0a21f9c3b16a8f107838c7b1bc351ac92cc332d5d891bad6d"}
Jul 28 12:34:15.300049 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.300031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp" event={"ID":"629f00bc-22c6-439d-9439-6113ad8ccef4","Type":"ContainerStarted","Data":"c56922778319242c9c43c166ddffd61c34da0e425e45ceae0deeebd2c0819ce3"}
Jul 28 12:34:15.404404 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.404323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:15.404544 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:15.404472    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:34:15.404544 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:15.404535    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls podName:b70dd332-8703-4163-a5f5-7aa2d722143b nodeName:}" failed. No retries permitted until 2026-07-28 12:34:16.404520061 +0000 UTC m=+43.764124644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls") pod "cluster-samples-operator-6748489656-jt876" (UID: "b70dd332-8703-4163-a5f5-7aa2d722143b") : secret "samples-operator-tls" not found
Jul 28 12:34:15.528641 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.528607    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:15.601523 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.601453    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:15.909719 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:15.909680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:15.909907 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:15.909868    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:34:15.909952 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:15.909932    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls podName:84a04ae8-3976-4608-92b8-26002d12fb16 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:23.90991405 +0000 UTC m=+51.269518637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls") pod "dns-default-dmbdk" (UID: "84a04ae8-3976-4608-92b8-26002d12fb16") : secret "dns-default-metrics-tls" not found
Jul 28 12:34:16.414615 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:16.414564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:16.414815 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:16.414762    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:34:16.414871 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:16.414842    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls podName:b70dd332-8703-4163-a5f5-7aa2d722143b nodeName:}" failed. No retries permitted until 2026-07-28 12:34:18.41482362 +0000 UTC m=+45.774428204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls") pod "cluster-samples-operator-6748489656-jt876" (UID: "b70dd332-8703-4163-a5f5-7aa2d722143b") : secret "samples-operator-tls" not found
Jul 28 12:34:17.020106 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:17.020062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:17.020570 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:17.020230    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:17.020570 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:17.020312    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls podName:54ff797e-a55b-4020-8e90-97d5bb21db66 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:25.020291033 +0000 UTC m=+52.379895621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7g7l" (UID: "54ff797e-a55b-4020-8e90-97d5bb21db66") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:18.310398 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.310363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" event={"ID":"39287a4b-97a2-495a-a4aa-e284ef517bf6","Type":"ContainerStarted","Data":"4c460faffe75b1ac70ffb59fd61f68d0ec3911f0fea2f8a0e78ca618b40727a4"}
Jul 28 12:34:18.310781 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.310572    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:18.312146 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.312117    2561 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-jhd5j container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.12:8443/readyz\": dial tcp 10.133.0.12:8443: connect: connection refused" start-of-body=
Jul 28 12:34:18.312255 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.312180    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" podUID="39287a4b-97a2-495a-a4aa-e284ef517bf6" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.12:8443/readyz\": dial tcp 10.133.0.12:8443: connect: connection refused"
Jul 28 12:34:18.312315 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.312260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp" event={"ID":"629f00bc-22c6-439d-9439-6113ad8ccef4","Type":"ContainerStarted","Data":"628075e4edf2e8df1656f91a471403baeeaa4f41fe0f8cb9e11413a10134ecc6"}
Jul 28 12:34:18.313636 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.313615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf" event={"ID":"56dbca26-1e1f-47ec-aa9c-88881434fa9e","Type":"ContainerStarted","Data":"fdd37d5d49e5dc91a9212dc6815227123503de18bb23e511cda33f6a751ee3b8"}
Jul 28 12:34:18.327491 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.327449    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" podStartSLOduration=1.392788787 podStartE2EDuration="4.327433717s" totalImagesPullingTime="2.93464493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:18.327397198 +0000 UTC m=+45.687001803" watchObservedRunningTime="2026-07-28 12:34:18.327433717 +0000 UTC m=+45.687038319"
Jul 28 12:34:18.343013 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.342960    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf" podStartSLOduration=1.348445244 podStartE2EDuration="4.342942968s" totalImagesPullingTime="2.994497724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:18.342506963 +0000 UTC m=+45.702111559" watchObservedRunningTime="2026-07-28 12:34:18.342942968 +0000 UTC m=+45.702547564"
Jul 28 12:34:18.357830 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.357718    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bvwkp" podStartSLOduration=1.403533314 podStartE2EDuration="4.357699626s" totalImagesPullingTime="2.954166312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:18.357355004 +0000 UTC m=+45.716959610" watchObservedRunningTime="2026-07-28 12:34:18.357699626 +0000 UTC m=+45.717304229"
Jul 28 12:34:18.432189 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.432149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:18.432594 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:18.432572    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:34:18.435758 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:18.435323    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls podName:b70dd332-8703-4163-a5f5-7aa2d722143b nodeName:}" failed. No retries permitted until 2026-07-28 12:34:22.435291103 +0000 UTC m=+49.794895706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls") pod "cluster-samples-operator-6748489656-jt876" (UID: "b70dd332-8703-4163-a5f5-7aa2d722143b") : secret "samples-operator-tls" not found
Jul 28 12:34:18.511365 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.511327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:18.613009 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:18.612931    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:19.317852 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:19.317829    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/0.log"
Jul 28 12:34:19.318312 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:19.317871    2561 generic.go:356] "Generic (PLEG): container finished" podID="39287a4b-97a2-495a-a4aa-e284ef517bf6" containerID="4c460faffe75b1ac70ffb59fd61f68d0ec3911f0fea2f8a0e78ca618b40727a4" exitCode=255
Jul 28 12:34:19.318312 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:19.317947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" event={"ID":"39287a4b-97a2-495a-a4aa-e284ef517bf6","Type":"ContainerDied","Data":"4c460faffe75b1ac70ffb59fd61f68d0ec3911f0fea2f8a0e78ca618b40727a4"}
Jul 28 12:34:19.318312 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:19.318210    2561 scope.go:117] "RemoveContainer" containerID="4c460faffe75b1ac70ffb59fd61f68d0ec3911f0fea2f8a0e78ca618b40727a4"
Jul 28 12:34:19.853984 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:19.853956    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:19.951353 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:19.951324    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:20.050441 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.050411    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:20.322879 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.322842    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6" event={"ID":"e19d70c4-98f0-4e28-9ced-f733a41d1dc4","Type":"ContainerStarted","Data":"3caf1e6cac16ba78f08d20edfeec559042a0b872c9bb355c964dbfe90a45358f"}
Jul 28 12:34:20.324770 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.324722    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:34:20.325155 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.325139    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/0.log"
Jul 28 12:34:20.325212 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.325174    2561 generic.go:356] "Generic (PLEG): container finished" podID="39287a4b-97a2-495a-a4aa-e284ef517bf6" containerID="8a56b530796cf6880e0859d53e81ecf9a90b91adb274d3e6ccaf86d24e0e8004" exitCode=255
Jul 28 12:34:20.325247 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.325204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" event={"ID":"39287a4b-97a2-495a-a4aa-e284ef517bf6","Type":"ContainerDied","Data":"8a56b530796cf6880e0859d53e81ecf9a90b91adb274d3e6ccaf86d24e0e8004"}
Jul 28 12:34:20.325280 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.325253    2561 scope.go:117] "RemoveContainer" containerID="4c460faffe75b1ac70ffb59fd61f68d0ec3911f0fea2f8a0e78ca618b40727a4"
Jul 28 12:34:20.325454 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.325428    2561 scope.go:117] "RemoveContainer" containerID="8a56b530796cf6880e0859d53e81ecf9a90b91adb274d3e6ccaf86d24e0e8004"
Jul 28 12:34:20.325619 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:20.325602    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-jhd5j_openshift-console-operator(39287a4b-97a2-495a-a4aa-e284ef517bf6)\"" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" podUID="39287a4b-97a2-495a-a4aa-e284ef517bf6"
Jul 28 12:34:20.358847 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.358792    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6" podStartSLOduration=2.042405295 podStartE2EDuration="6.358724854s" totalImagesPullingTime="4.316319559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:20.341823491 +0000 UTC m=+47.701428095" watchObservedRunningTime="2026-07-28 12:34:20.358724854 +0000 UTC m=+47.718329460"
Jul 28 12:34:20.619672 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.619592    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:20.719940 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:20.719909    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:21.305265 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:21.305214    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:21.328866 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:21.328844    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:34:21.329216 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:21.329204    2561 scope.go:117] "RemoveContainer" containerID="8a56b530796cf6880e0859d53e81ecf9a90b91adb274d3e6ccaf86d24e0e8004"
Jul 28 12:34:21.329370 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:21.329351    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-jhd5j_openshift-console-operator(39287a4b-97a2-495a-a4aa-e284ef517bf6)\"" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" podUID="39287a4b-97a2-495a-a4aa-e284ef517bf6"
Jul 28 12:34:21.404063 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:21.404030    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:21.503965 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:21.503926    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:21.764469 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:21.764366    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:21.865426 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:21.865388    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:22.018127 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:22.018051    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:22.112576 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:22.112543    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:22.420152 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:22.420116    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:22.468650 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:22.468615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:22.468850 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:22.468757    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:34:22.468850 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:22.468823    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls podName:b70dd332-8703-4163-a5f5-7aa2d722143b nodeName:}" failed. No retries permitted until 2026-07-28 12:34:30.46880364 +0000 UTC m=+57.828408222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls") pod "cluster-samples-operator-6748489656-jt876" (UID: "b70dd332-8703-4163-a5f5-7aa2d722143b") : secret "samples-operator-tls" not found
Jul 28 12:34:22.524605 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:22.524576    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:23.621292 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.621259    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-np984"]
Jul 28 12:34:23.642361 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.642337    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-np984"]
Jul 28 12:34:23.642492 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.642426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.645239 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.645214    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 12:34:23.645239 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.645223    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 12:34:23.645509 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.645492    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 12:34:23.646487 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.646467    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 12:34:23.646592 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.646536    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xdlvr\""
Jul 28 12:34:23.661853 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.661827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:23.727155 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.727129    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:23.780291 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.780263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8sjn\" (UniqueName: \"kubernetes.io/projected/e62e2a96-ff83-4099-b8b2-b34c3bab3949-kube-api-access-m8sjn\") pod \"service-ca-658d7b7c78-np984\" (UID: \"e62e2a96-ff83-4099-b8b2-b34c3bab3949\") " pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.780405 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.780331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e62e2a96-ff83-4099-b8b2-b34c3bab3949-signing-key\") pod \"service-ca-658d7b7c78-np984\" (UID: \"e62e2a96-ff83-4099-b8b2-b34c3bab3949\") " pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.780405 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.780354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e62e2a96-ff83-4099-b8b2-b34c3bab3949-signing-cabundle\") pod \"service-ca-658d7b7c78-np984\" (UID: \"e62e2a96-ff83-4099-b8b2-b34c3bab3949\") " pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.881619 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.881548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e62e2a96-ff83-4099-b8b2-b34c3bab3949-signing-key\") pod \"service-ca-658d7b7c78-np984\" (UID: \"e62e2a96-ff83-4099-b8b2-b34c3bab3949\") " pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.881619 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.881578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e62e2a96-ff83-4099-b8b2-b34c3bab3949-signing-cabundle\") pod \"service-ca-658d7b7c78-np984\" (UID: \"e62e2a96-ff83-4099-b8b2-b34c3bab3949\") " pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.881798 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.881651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8sjn\" (UniqueName: \"kubernetes.io/projected/e62e2a96-ff83-4099-b8b2-b34c3bab3949-kube-api-access-m8sjn\") pod \"service-ca-658d7b7c78-np984\" (UID: \"e62e2a96-ff83-4099-b8b2-b34c3bab3949\") " pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.882352 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.882327    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e62e2a96-ff83-4099-b8b2-b34c3bab3949-signing-cabundle\") pod \"service-ca-658d7b7c78-np984\" (UID: \"e62e2a96-ff83-4099-b8b2-b34c3bab3949\") " pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.884098 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.884082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e62e2a96-ff83-4099-b8b2-b34c3bab3949-signing-key\") pod \"service-ca-658d7b7c78-np984\" (UID: \"e62e2a96-ff83-4099-b8b2-b34c3bab3949\") " pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.891102 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.891079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8sjn\" (UniqueName: \"kubernetes.io/projected/e62e2a96-ff83-4099-b8b2-b34c3bab3949-kube-api-access-m8sjn\") pod \"service-ca-658d7b7c78-np984\" (UID: \"e62e2a96-ff83-4099-b8b2-b34c3bab3949\") " pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.951470 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.951439    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-np984"
Jul 28 12:34:23.982486 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:23.982456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:23.982610 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:23.982589    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:34:23.982681 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:23.982670    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls podName:84a04ae8-3976-4608-92b8-26002d12fb16 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:39.98264746 +0000 UTC m=+67.342252063 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls") pod "dns-default-dmbdk" (UID: "84a04ae8-3976-4608-92b8-26002d12fb16") : secret "dns-default-metrics-tls" not found
Jul 28 12:34:24.091996 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:24.091968    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-np984"]
Jul 28 12:34:24.094891 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:24.094860    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62e2a96_ff83_4099_b8b2_b34c3bab3949.slice/crio-83b71ab46996acf7dae77c912082f85df35287b43e23586f02cede374a956682 WatchSource:0}: Error finding container 83b71ab46996acf7dae77c912082f85df35287b43e23586f02cede374a956682: Status 404 returned error can't find the container with id 83b71ab46996acf7dae77c912082f85df35287b43e23586f02cede374a956682
Jul 28 12:34:24.338556 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:24.338520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-np984" event={"ID":"e62e2a96-ff83-4099-b8b2-b34c3bab3949","Type":"ContainerStarted","Data":"8e179483a0d075f32ab90946c8b137246e70520c85cbd48a1a7b05ef9b72fefc"}
Jul 28 12:34:24.338727 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:24.338574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-np984" event={"ID":"e62e2a96-ff83-4099-b8b2-b34c3bab3949","Type":"ContainerStarted","Data":"83b71ab46996acf7dae77c912082f85df35287b43e23586f02cede374a956682"}
Jul 28 12:34:24.359812 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:24.359758    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-np984" podStartSLOduration=1.359724324 podStartE2EDuration="1.359724324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:24.359147848 +0000 UTC m=+51.718752453" watchObservedRunningTime="2026-07-28 12:34:24.359724324 +0000 UTC m=+51.719328929"
Jul 28 12:34:24.767775 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:24.767718    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:24.983366 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:24.983325    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:24.983756 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:24.983718    2561 scope.go:117] "RemoveContainer" containerID="8a56b530796cf6880e0859d53e81ecf9a90b91adb274d3e6ccaf86d24e0e8004"
Jul 28 12:34:24.983915 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:24.983895    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-jhd5j_openshift-console-operator(39287a4b-97a2-495a-a4aa-e284ef517bf6)\"" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" podUID="39287a4b-97a2-495a-a4aa-e284ef517bf6"
Jul 28 12:34:25.091602 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:25.091568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:25.091822 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:25.091773    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:25.091902 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:25.091857    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls podName:54ff797e-a55b-4020-8e90-97d5bb21db66 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:41.09183401 +0000 UTC m=+68.451438598 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7g7l" (UID: "54ff797e-a55b-4020-8e90-97d5bb21db66") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:34:25.814363 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:25.814313    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:25.915987 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:25.915943    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:26.254361 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:26.254262    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:26.446147 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:26.446114    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:26.539046 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:26.539014    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:27.512851 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:27.512817    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:27.709049 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:27.709016    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:28.088900 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:28.088868    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:28.171646 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:28.171614    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:28.311163 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:28.311126    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:28.311540 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:28.311525    2561 scope.go:117] "RemoveContainer" containerID="8a56b530796cf6880e0859d53e81ecf9a90b91adb274d3e6ccaf86d24e0e8004"
Jul 28 12:34:28.311705 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:28.311686    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-jhd5j_openshift-console-operator(39287a4b-97a2-495a-a4aa-e284ef517bf6)\"" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" podUID="39287a4b-97a2-495a-a4aa-e284ef517bf6"
Jul 28 12:34:29.554496 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:29.554462    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:29.620218 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:29.620189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:29.719094 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:29.719060    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:29.901166 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:29.901082    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:30.102278 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:30.102248    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:30.517641 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:30.517609    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:30.539902 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:30.539870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:30.542589 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:30.542554    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b70dd332-8703-4163-a5f5-7aa2d722143b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-jt876\" (UID: \"b70dd332-8703-4163-a5f5-7aa2d722143b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:30.600554 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:30.600509    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"
Jul 28 12:34:30.620953 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:30.620928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:30.723677 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:30.723652    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:30.762471 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:30.762408    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876"]
Jul 28 12:34:31.099152 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:31.099063    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:31.365498 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:31.365422    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876" event={"ID":"b70dd332-8703-4163-a5f5-7aa2d722143b","Type":"ContainerStarted","Data":"836e37187a372ec6495126c4f63b3c6e7731c919fb52249d77e397284c5327fc"}
Jul 28 12:34:31.505025 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:31.504987    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:31.908273 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:31.908244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:32.504902 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:32.504855    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:32.946057 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:32.946022    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:33.046925 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:33.046892    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:33.146991 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:33.146959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:33.372315 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:33.372282    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876" event={"ID":"b70dd332-8703-4163-a5f5-7aa2d722143b","Type":"ContainerStarted","Data":"51c686d92e33c0ef78f91cbf54712acafc42ea6d04da53db7f1bfe9732b77f9b"}
Jul 28 12:34:33.541214 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:33.541184    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:34.099485 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:34.099433    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:34.270072 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:34.270033    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbd9x"
Jul 28 12:34:34.377753 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:34.377651    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876" event={"ID":"b70dd332-8703-4163-a5f5-7aa2d722143b","Type":"ContainerStarted","Data":"f39b49853234ba402e46be88994e431d3c4b2718e9bd6b62661f72a1231f3a5f"}
Jul 28 12:34:34.397267 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:34.397207    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-jt876" podStartSLOduration=17.999637437 podStartE2EDuration="20.397190563s" totalImagesPullingTime="2.397553126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:34.396629499 +0000 UTC m=+61.756234105" watchObservedRunningTime="2026-07-28 12:34:34.397190563 +0000 UTC m=+61.756795168"
Jul 28 12:34:34.502192 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:34.502160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:34.694003 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:34.693915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:35.098201 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:35.098140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:35.716779 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:35.716677    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:35.817153 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:35.817114    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:35.920705 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:35.920671    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:36.204775 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:36.204715    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:36.549070 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:36.549038    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:38.783179 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:38.783151    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:38.823515 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:38.823478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:34:38.826175 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:38.826158    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:34:38.836925 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:38.836898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18584de2-6d71-420c-a072-9b4ff6950071-metrics-certs\") pod \"network-metrics-daemon-tjcgm\" (UID: \"18584de2-6d71-420c-a072-9b4ff6950071\") " pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:34:38.923842 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:38.923810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:34:38.924882 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:38.924822    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:38.926064 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:38.926050    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:34:38.937348 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:38.937325    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14ee1edf-eb79-49a6-b644-7fb9f2bf93d0-original-pull-secret\") pod \"global-pull-secret-syncer-h72fp\" (UID: \"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0\") " pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:34:39.012137 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.012107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7n8fl\""
Jul 28 12:34:39.020472 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.020447    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjcgm"
Jul 28 12:34:39.023436 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.023409    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72fp"
Jul 28 12:34:39.024964 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.024939    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhjl\" (UniqueName: \"kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl\") pod \"network-check-target-zn6gv\" (UID: \"04e859b2-becc-46c1-a587-64e662d550dd\") " pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:34:39.027411 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.027393    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:34:39.038008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.037986    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:34:39.048917 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.048892    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhjl\" (UniqueName: \"kubernetes.io/projected/04e859b2-becc-46c1-a587-64e662d550dd-kube-api-access-qlhjl\") pod \"network-check-target-zn6gv\" (UID: \"04e859b2-becc-46c1-a587-64e662d550dd\") " pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:34:39.073577 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.073542    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:39.167877 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.167848    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tjcgm"]
Jul 28 12:34:39.170743 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:39.170705    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18584de2_6d71_420c_a072_9b4ff6950071.slice/crio-0f62aa48469d8c2e5400627ed8482c4ad4d8bf1feea4673c4c0189fdc73b5641 WatchSource:0}: Error finding container 0f62aa48469d8c2e5400627ed8482c4ad4d8bf1feea4673c4c0189fdc73b5641: Status 404 returned error can't find the container with id 0f62aa48469d8c2e5400627ed8482c4ad4d8bf1feea4673c4c0189fdc73b5641
Jul 28 12:34:39.182282 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.182252    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h72fp"]
Jul 28 12:34:39.186198 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:39.186172    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ee1edf_eb79_49a6_b644_7fb9f2bf93d0.slice/crio-ddcf9890c4b26bed60de1ac65acc0fd1075a9329ad1e9390356196d0ecc8d059 WatchSource:0}: Error finding container ddcf9890c4b26bed60de1ac65acc0fd1075a9329ad1e9390356196d0ecc8d059: Status 404 returned error can't find the container with id ddcf9890c4b26bed60de1ac65acc0fd1075a9329ad1e9390356196d0ecc8d059
Jul 28 12:34:39.319653 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.319563    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jpxcx\""
Jul 28 12:34:39.328043 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.328021    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:34:39.394780 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.394716    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h72fp" event={"ID":"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0","Type":"ContainerStarted","Data":"ddcf9890c4b26bed60de1ac65acc0fd1075a9329ad1e9390356196d0ecc8d059"}
Jul 28 12:34:39.396945 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.396453    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tjcgm" event={"ID":"18584de2-6d71-420c-a072-9b4ff6950071","Type":"ContainerStarted","Data":"0f62aa48469d8c2e5400627ed8482c4ad4d8bf1feea4673c4c0189fdc73b5641"}
Jul 28 12:34:39.478100 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:39.478073    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zn6gv"]
Jul 28 12:34:39.480444 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:39.480414    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e859b2_becc_46c1_a587_64e662d550dd.slice/crio-56f92eb81263698301193f3a66840f8a83904581eea636a954edc0477bd8b98f WatchSource:0}: Error finding container 56f92eb81263698301193f3a66840f8a83904581eea636a954edc0477bd8b98f: Status 404 returned error can't find the container with id 56f92eb81263698301193f3a66840f8a83904581eea636a954edc0477bd8b98f
Jul 28 12:34:40.032399 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:40.032365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:40.035097 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:40.035070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84a04ae8-3976-4608-92b8-26002d12fb16-metrics-tls\") pod \"dns-default-dmbdk\" (UID: \"84a04ae8-3976-4608-92b8-26002d12fb16\") " pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:40.246025 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:40.245987    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g8tcp\""
Jul 28 12:34:40.254059 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:40.254025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:40.401488 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:40.401448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zn6gv" event={"ID":"04e859b2-becc-46c1-a587-64e662d550dd","Type":"ContainerStarted","Data":"56f92eb81263698301193f3a66840f8a83904581eea636a954edc0477bd8b98f"}
Jul 28 12:34:40.426577 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:40.426531    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dmbdk"]
Jul 28 12:34:40.580776 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:40.580673    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a04ae8_3976_4608_92b8_26002d12fb16.slice/crio-383f2fb909f36b0703a3a844d86d3dc9f950529a70d43fb368e0f7399c1ab04a WatchSource:0}: Error finding container 383f2fb909f36b0703a3a844d86d3dc9f950529a70d43fb368e0f7399c1ab04a: Status 404 returned error can't find the container with id 383f2fb909f36b0703a3a844d86d3dc9f950529a70d43fb368e0f7399c1ab04a
Jul 28 12:34:41.142924 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:41.142773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:41.147516 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:41.147481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54ff797e-a55b-4020-8e90-97d5bb21db66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7g7l\" (UID: \"54ff797e-a55b-4020-8e90-97d5bb21db66\") " pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:41.408344 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:41.408252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tjcgm" event={"ID":"18584de2-6d71-420c-a072-9b4ff6950071","Type":"ContainerStarted","Data":"735f41db6bad5266e90f4d4b2947a1aba3ec12fe261777a201b153ada866902c"}
Jul 28 12:34:41.408344 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:41.408326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tjcgm" event={"ID":"18584de2-6d71-420c-a072-9b4ff6950071","Type":"ContainerStarted","Data":"77c322f0e0b9abc96f1b4098bfc48710c911c5a1cfa34b4e9d6dd17e3b9048e9"}
Jul 28 12:34:41.410121 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:41.410049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dmbdk" event={"ID":"84a04ae8-3976-4608-92b8-26002d12fb16","Type":"ContainerStarted","Data":"383f2fb909f36b0703a3a844d86d3dc9f950529a70d43fb368e0f7399c1ab04a"}
Jul 28 12:34:41.426895 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:41.426680    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nnkkf\""
Jul 28 12:34:41.428690 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:41.427517    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tjcgm" podStartSLOduration=66.99161771 podStartE2EDuration="1m8.427500465s" totalImagesPullingTime="1.435882755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:41.425624776 +0000 UTC m=+68.785229382" watchObservedRunningTime="2026-07-28 12:34:41.427500465 +0000 UTC m=+68.787105071"
Jul 28 12:34:41.434439 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:41.434412    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b7g7l"
Jul 28 12:34:41.625539 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:41.625457    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b7g7l"]
Jul 28 12:34:42.415167 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:42.415117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7g7l" event={"ID":"54ff797e-a55b-4020-8e90-97d5bb21db66","Type":"ContainerStarted","Data":"dfad8aa7c627c040e5d2eca27e0e391bcbb98eb83a8a516444c0b9e092a8bcd5"}
Jul 28 12:34:43.636177 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:43.636141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:43.697968 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:43.697930    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:44.097318 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:44.097280    2561 scope.go:117] "RemoveContainer" containerID="8a56b530796cf6880e0859d53e81ecf9a90b91adb274d3e6ccaf86d24e0e8004"
Jul 28 12:34:45.005131 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.005086    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"]
Jul 28 12:34:45.009555 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.009454    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"]
Jul 28 12:34:45.009785 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.009712    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.017659 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.015563    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 12:34:45.017659 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.016357    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 12:34:45.017659 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.016665    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 12:34:45.017659 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.017031    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 12:34:45.020783 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.020590    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cf5c85c5f-nc4vj"]
Jul 28 12:34:45.021890 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.021826    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"
Jul 28 12:34:45.028807 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.027921    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 12:34:45.028807 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.028162    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-z7wgm\""
Jul 28 12:34:45.030427 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.030401    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"]
Jul 28 12:34:45.031937 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.031910    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"]
Jul 28 12:34:45.032184 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.030663    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.043532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.041702    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 12:34:45.043532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.042093    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpxsc\""
Jul 28 12:34:45.043532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.042322    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 12:34:45.043532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.042576    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 12:34:45.045668 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.045638    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cf5c85c5f-nc4vj"]
Jul 28 12:34:45.049540 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.049491    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 12:34:45.076618 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.076592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93fd96f1-c30b-4cbb-b3d7-9385c631443c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dd9d75594-jhr99\" (UID: \"93fd96f1-c30b-4cbb-b3d7-9385c631443c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"
Jul 28 12:34:45.076725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.076663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-certificates\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.076725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.076700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxchj\" (UniqueName: \"kubernetes.io/projected/93fd96f1-c30b-4cbb-b3d7-9385c631443c-kube-api-access-zxchj\") pod \"managed-serviceaccount-addon-agent-6dd9d75594-jhr99\" (UID: \"93fd96f1-c30b-4cbb-b3d7-9385c631443c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"
Jul 28 12:34:45.076842 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.076773    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a3e8e902-7442-4ef7-9545-9a976a21ceb1-ca-trust-extracted\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.076918 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.076902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75b3692f-d5f5-4012-83b1-9fee03901248-tmp\") pod \"klusterlet-addon-workmgr-7db697754b-c6lp7\" (UID: \"75b3692f-d5f5-4012-83b1-9fee03901248\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.076994 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.076978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-tls\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.077040 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.077008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w94nj\" (UniqueName: \"kubernetes.io/projected/75b3692f-d5f5-4012-83b1-9fee03901248-kube-api-access-w94nj\") pod \"klusterlet-addon-workmgr-7db697754b-c6lp7\" (UID: \"75b3692f-d5f5-4012-83b1-9fee03901248\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.077096 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.077082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/75b3692f-d5f5-4012-83b1-9fee03901248-klusterlet-config\") pod \"klusterlet-addon-workmgr-7db697754b-c6lp7\" (UID: \"75b3692f-d5f5-4012-83b1-9fee03901248\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.077170 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.077149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-trusted-ca\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.077252 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.077229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-installation-pull-secrets\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.077324 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.077307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-image-registry-private-configuration\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.077399 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.077363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krgvg\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-kube-api-access-krgvg\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.077456 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.077434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-bound-sa-token\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.093937 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.093859    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cf5c85c5f-nc4vj"]
Jul 28 12:34:45.108754 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.107540    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk"]
Jul 28 12:34:45.112339 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.112307    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk"
Jul 28 12:34:45.114832 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.114679    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-55pgp\""
Jul 28 12:34:45.119175 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:34:45.119123    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-krgvg registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj" podUID="a3e8e902-7442-4ef7-9545-9a976a21ceb1"
Jul 28 12:34:45.126369 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.124519    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk"]
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-trusted-ca\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-installation-pull-secrets\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-image-registry-private-configuration\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgvg\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-kube-api-access-krgvg\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-bound-sa-token\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93fd96f1-c30b-4cbb-b3d7-9385c631443c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dd9d75594-jhr99\" (UID: \"93fd96f1-c30b-4cbb-b3d7-9385c631443c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-certificates\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxchj\" (UniqueName: \"kubernetes.io/projected/93fd96f1-c30b-4cbb-b3d7-9385c631443c-kube-api-access-zxchj\") pod \"managed-serviceaccount-addon-agent-6dd9d75594-jhr99\" (UID: \"93fd96f1-c30b-4cbb-b3d7-9385c631443c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a3e8e902-7442-4ef7-9545-9a976a21ceb1-ca-trust-extracted\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178517    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jkfz\" (UniqueName: \"kubernetes.io/projected/c231cdd2-c1fd-42fe-a6ed-1cde78591cc1-kube-api-access-5jkfz\") pod \"network-check-source-7d88cb55d8-nqdtk\" (UID: \"c231cdd2-c1fd-42fe-a6ed-1cde78591cc1\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75b3692f-d5f5-4012-83b1-9fee03901248-tmp\") pod \"klusterlet-addon-workmgr-7db697754b-c6lp7\" (UID: \"75b3692f-d5f5-4012-83b1-9fee03901248\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178584    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-tls\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w94nj\" (UniqueName: \"kubernetes.io/projected/75b3692f-d5f5-4012-83b1-9fee03901248-kube-api-access-w94nj\") pod \"klusterlet-addon-workmgr-7db697754b-c6lp7\" (UID: \"75b3692f-d5f5-4012-83b1-9fee03901248\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.179200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.178646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/75b3692f-d5f5-4012-83b1-9fee03901248-klusterlet-config\") pod \"klusterlet-addon-workmgr-7db697754b-c6lp7\" (UID: \"75b3692f-d5f5-4012-83b1-9fee03901248\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.180065 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.179353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-trusted-ca\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.180772 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.180392    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-certificates\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.180772 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.180591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75b3692f-d5f5-4012-83b1-9fee03901248-tmp\") pod \"klusterlet-addon-workmgr-7db697754b-c6lp7\" (UID: \"75b3692f-d5f5-4012-83b1-9fee03901248\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.180772 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.180696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a3e8e902-7442-4ef7-9545-9a976a21ceb1-ca-trust-extracted\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.182630 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.182602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-installation-pull-secrets\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.183785 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.183688    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-image-registry-private-configuration\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.183882 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.183867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/75b3692f-d5f5-4012-83b1-9fee03901248-klusterlet-config\") pod \"klusterlet-addon-workmgr-7db697754b-c6lp7\" (UID: \"75b3692f-d5f5-4012-83b1-9fee03901248\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.184848 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.184804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-tls\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.187250 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.187197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93fd96f1-c30b-4cbb-b3d7-9385c631443c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dd9d75594-jhr99\" (UID: \"93fd96f1-c30b-4cbb-b3d7-9385c631443c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"
Jul 28 12:34:45.195550 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.195523    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgvg\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-kube-api-access-krgvg\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.195972 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.195929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-bound-sa-token\") pod \"image-registry-cf5c85c5f-nc4vj\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") " pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.196476 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.196446    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxchj\" (UniqueName: \"kubernetes.io/projected/93fd96f1-c30b-4cbb-b3d7-9385c631443c-kube-api-access-zxchj\") pod \"managed-serviceaccount-addon-agent-6dd9d75594-jhr99\" (UID: \"93fd96f1-c30b-4cbb-b3d7-9385c631443c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"
Jul 28 12:34:45.197520 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.197497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w94nj\" (UniqueName: \"kubernetes.io/projected/75b3692f-d5f5-4012-83b1-9fee03901248-kube-api-access-w94nj\") pod \"klusterlet-addon-workmgr-7db697754b-c6lp7\" (UID: \"75b3692f-d5f5-4012-83b1-9fee03901248\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.207845 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.207817    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:45.279407 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.279370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jkfz\" (UniqueName: \"kubernetes.io/projected/c231cdd2-c1fd-42fe-a6ed-1cde78591cc1-kube-api-access-5jkfz\") pod \"network-check-source-7d88cb55d8-nqdtk\" (UID: \"c231cdd2-c1fd-42fe-a6ed-1cde78591cc1\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk"
Jul 28 12:34:45.288150 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.288114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jkfz\" (UniqueName: \"kubernetes.io/projected/c231cdd2-c1fd-42fe-a6ed-1cde78591cc1-kube-api-access-5jkfz\") pod \"network-check-source-7d88cb55d8-nqdtk\" (UID: \"c231cdd2-c1fd-42fe-a6ed-1cde78591cc1\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk"
Jul 28 12:34:45.299883 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.299820    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:45.373575 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.373318    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:45.402624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.402338    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"
Jul 28 12:34:45.404713 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.404676    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:45.431068 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.430191    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zn6gv" event={"ID":"04e859b2-becc-46c1-a587-64e662d550dd","Type":"ContainerStarted","Data":"c4631f72843bd46905c8561da73cae504b2b71185579cea4d52e6c0bf20c20a3"}
Jul 28 12:34:45.431068 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.431014    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:34:45.436756 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.436700    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dmbdk" event={"ID":"84a04ae8-3976-4608-92b8-26002d12fb16","Type":"ContainerStarted","Data":"fb3e8eb207ceabb8e65d1d516d0cf14de0b3f99f108156efdbb6dfd10c9e330b"}
Jul 28 12:34:45.436910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.436788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dmbdk" event={"ID":"84a04ae8-3976-4608-92b8-26002d12fb16","Type":"ContainerStarted","Data":"644c4e43bdca01d139c805223b63ccf5ae6a0411f05a96ac16f824ed4cc8c009"}
Jul 28 12:34:45.437378 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.437359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:45.440126 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.440053    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:34:45.440253 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.440144    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j" event={"ID":"39287a4b-97a2-495a-a4aa-e284ef517bf6","Type":"ContainerStarted","Data":"4b12ff9ec017f3d7d9add60d97dfbcc13ed5373558a7165ca2dddb9e4d71dc92"}
Jul 28 12:34:45.441294 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.440724    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:45.443378 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.443147    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.444183 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.443723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h72fp" event={"ID":"14ee1edf-eb79-49a6-b644-7fb9f2bf93d0","Type":"ContainerStarted","Data":"2abf06f5c5278c2494972b8d83f2d724678257564d51ad46288833b6b39e8899"}
Jul 28 12:34:45.446690 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.446015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk"
Jul 28 12:34:45.452875 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.452819    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zn6gv" podStartSLOduration=66.982208078 podStartE2EDuration="1m12.45279937s" totalImagesPullingTime="5.470591292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:45.450587792 +0000 UTC m=+72.810192399" watchObservedRunningTime="2026-07-28 12:34:45.45279937 +0000 UTC m=+72.812404016"
Jul 28 12:34:45.458768 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.458056    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:45.475790 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.475706    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h72fp" podStartSLOduration=66.711378254 podStartE2EDuration="1m12.475683077s" totalImagesPullingTime="5.764304823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:45.4752833 +0000 UTC m=+72.834887895" watchObservedRunningTime="2026-07-28 12:34:45.475683077 +0000 UTC m=+72.835287686"
Jul 28 12:34:45.481003 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.480972    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-trusted-ca\") pod \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") "
Jul 28 12:34:45.481159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.481026    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a3e8e902-7442-4ef7-9545-9a976a21ceb1-ca-trust-extracted\") pod \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") "
Jul 28 12:34:45.481159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.481072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-image-registry-private-configuration\") pod \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") "
Jul 28 12:34:45.481159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.481105    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-certificates\") pod \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") "
Jul 28 12:34:45.481159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.481155    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-bound-sa-token\") pod \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") "
Jul 28 12:34:45.481776 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.481199    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-installation-pull-secrets\") pod \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") "
Jul 28 12:34:45.481776 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.481230    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krgvg\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-kube-api-access-krgvg\") pod \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") "
Jul 28 12:34:45.481776 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.481257    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-tls\") pod \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\" (UID: \"a3e8e902-7442-4ef7-9545-9a976a21ceb1\") "
Jul 28 12:34:45.489114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.489082    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a3e8e902-7442-4ef7-9545-9a976a21ceb1" (UID: "a3e8e902-7442-4ef7-9545-9a976a21ceb1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:34:45.490264 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.490224    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e8e902-7442-4ef7-9545-9a976a21ceb1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a3e8e902-7442-4ef7-9545-9a976a21ceb1" (UID: "a3e8e902-7442-4ef7-9545-9a976a21ceb1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:34:45.490788 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.490751    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a3e8e902-7442-4ef7-9545-9a976a21ceb1" (UID: "a3e8e902-7442-4ef7-9545-9a976a21ceb1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:34:45.494354 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.494300    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a3e8e902-7442-4ef7-9545-9a976a21ceb1" (UID: "a3e8e902-7442-4ef7-9545-9a976a21ceb1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:34:45.498318 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.498270    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a3e8e902-7442-4ef7-9545-9a976a21ceb1" (UID: "a3e8e902-7442-4ef7-9545-9a976a21ceb1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:34:45.498592 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.498560    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-kube-api-access-krgvg" (OuterVolumeSpecName: "kube-api-access-krgvg") pod "a3e8e902-7442-4ef7-9545-9a976a21ceb1" (UID: "a3e8e902-7442-4ef7-9545-9a976a21ceb1"). InnerVolumeSpecName "kube-api-access-krgvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:34:45.498895 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.498861    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a3e8e902-7442-4ef7-9545-9a976a21ceb1" (UID: "a3e8e902-7442-4ef7-9545-9a976a21ceb1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:34:45.501149 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.501114    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a3e8e902-7442-4ef7-9545-9a976a21ceb1" (UID: "a3e8e902-7442-4ef7-9545-9a976a21ceb1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:34:45.515562 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.514599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:45.534995 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.534808    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dmbdk" podStartSLOduration=33.194332242 podStartE2EDuration="37.53478257s" totalImagesPullingTime="4.340450328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:45.513982695 +0000 UTC m=+72.873587300" watchObservedRunningTime="2026-07-28 12:34:45.53478257 +0000 UTC m=+72.894387179"
Jul 28 12:34:45.573994 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.573957    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"]
Jul 28 12:34:45.581304 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:45.581222    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b3692f_d5f5_4012_83b1_9fee03901248.slice/crio-954a00a0314c8711eb55cbba4f7f1a23ee2b039a609b560fd9da9f02fb7e37d3 WatchSource:0}: Error finding container 954a00a0314c8711eb55cbba4f7f1a23ee2b039a609b560fd9da9f02fb7e37d3: Status 404 returned error can't find the container with id 954a00a0314c8711eb55cbba4f7f1a23ee2b039a609b560fd9da9f02fb7e37d3
Jul 28 12:34:45.582175 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.582150    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a3e8e902-7442-4ef7-9545-9a976a21ceb1-ca-trust-extracted\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:34:45.582175 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.582182    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-image-registry-private-configuration\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:34:45.582346 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.582198    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-certificates\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:34:45.582346 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.582243    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-bound-sa-token\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:34:45.582346 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.582257    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a3e8e902-7442-4ef7-9545-9a976a21ceb1-installation-pull-secrets\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:34:45.582346 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.582270    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krgvg\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-kube-api-access-krgvg\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:34:45.582346 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.582283    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a3e8e902-7442-4ef7-9545-9a976a21ceb1-registry-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:34:45.582346 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.582310    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3e8e902-7442-4ef7-9545-9a976a21ceb1-trusted-ca\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:34:45.606416 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.606331    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99"]
Jul 28 12:34:45.627970 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:45.627942    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93fd96f1_c30b_4cbb_b3d7_9385c631443c.slice/crio-11a6bb56fd57b87586aa87917d29ee5d21ca7bdffc0d8389969f786acb2716d7 WatchSource:0}: Error finding container 11a6bb56fd57b87586aa87917d29ee5d21ca7bdffc0d8389969f786acb2716d7: Status 404 returned error can't find the container with id 11a6bb56fd57b87586aa87917d29ee5d21ca7bdffc0d8389969f786acb2716d7
Jul 28 12:34:45.667936 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.667705    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk"]
Jul 28 12:34:45.670910 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:45.670880    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc231cdd2_c1fd_42fe_a6ed_1cde78591cc1.slice/crio-ad6353508eae8c09bd7fb7fa1ee76dde132107cbb5477aff2190dd2ce95c6780 WatchSource:0}: Error finding container ad6353508eae8c09bd7fb7fa1ee76dde132107cbb5477aff2190dd2ce95c6780: Status 404 returned error can't find the container with id ad6353508eae8c09bd7fb7fa1ee76dde132107cbb5477aff2190dd2ce95c6780
Jul 28 12:34:45.941670 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:45.941580    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-jhd5j"
Jul 28 12:34:46.131917 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.131881    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-w58tj"]
Jul 28 12:34:46.135518 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.135490    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-w58tj"
Jul 28 12:34:46.138381 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.138225    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-qdpth\""
Jul 28 12:34:46.139312 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.138487    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:34:46.139312 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.138504    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:34:46.139312 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.138721    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:46.150129 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.150084    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-w58tj"]
Jul 28 12:34:46.187511 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.187477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjwv\" (UniqueName: \"kubernetes.io/projected/14b46c60-e496-4c48-baa3-1780844ca48d-kube-api-access-msjwv\") pod \"downloads-7c97dc4555-w58tj\" (UID: \"14b46c60-e496-4c48-baa3-1780844ca48d\") " pod="openshift-console/downloads-7c97dc4555-w58tj"
Jul 28 12:34:46.240390 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.240312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:46.287979 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.287946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjwv\" (UniqueName: \"kubernetes.io/projected/14b46c60-e496-4c48-baa3-1780844ca48d-kube-api-access-msjwv\") pod \"downloads-7c97dc4555-w58tj\" (UID: \"14b46c60-e496-4c48-baa3-1780844ca48d\") " pod="openshift-console/downloads-7c97dc4555-w58tj"
Jul 28 12:34:46.301836 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.301809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjwv\" (UniqueName: \"kubernetes.io/projected/14b46c60-e496-4c48-baa3-1780844ca48d-kube-api-access-msjwv\") pod \"downloads-7c97dc4555-w58tj\" (UID: \"14b46c60-e496-4c48-baa3-1780844ca48d\") " pod="openshift-console/downloads-7c97dc4555-w58tj"
Jul 28 12:34:46.353383 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.353352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:46.448196 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.447948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-w58tj"
Jul 28 12:34:46.450093 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.450060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99" event={"ID":"93fd96f1-c30b-4cbb-b3d7-9385c631443c","Type":"ContainerStarted","Data":"11a6bb56fd57b87586aa87917d29ee5d21ca7bdffc0d8389969f786acb2716d7"}
Jul 28 12:34:46.451567 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.451526    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7" event={"ID":"75b3692f-d5f5-4012-83b1-9fee03901248","Type":"ContainerStarted","Data":"954a00a0314c8711eb55cbba4f7f1a23ee2b039a609b560fd9da9f02fb7e37d3"}
Jul 28 12:34:46.454148 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.454119    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk" event={"ID":"c231cdd2-c1fd-42fe-a6ed-1cde78591cc1","Type":"ContainerStarted","Data":"3d4ff641f6b1a6362b040acdb5237989ae398aba0e5b7dfd01ee6c7c708c2721"}
Jul 28 12:34:46.454240 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.454163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk" event={"ID":"c231cdd2-c1fd-42fe-a6ed-1cde78591cc1","Type":"ContainerStarted","Data":"ad6353508eae8c09bd7fb7fa1ee76dde132107cbb5477aff2190dd2ce95c6780"}
Jul 28 12:34:46.454518 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.454499    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cf5c85c5f-nc4vj"
Jul 28 12:34:46.472097 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.472049    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nqdtk" podStartSLOduration=1.472028978 podStartE2EDuration="1.472028978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:46.470190764 +0000 UTC m=+73.829795368" watchObservedRunningTime="2026-07-28 12:34:46.472028978 +0000 UTC m=+73.831633581"
Jul 28 12:34:46.502121 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.502026    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cf5c85c5f-nc4vj"]
Jul 28 12:34:46.506172 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.506139    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cf5c85c5f-nc4vj"]
Jul 28 12:34:46.605984 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:46.605948    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-w58tj"]
Jul 28 12:34:46.611867 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:34:46.611842    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b46c60_e496_4c48_baa3_1780844ca48d.slice/crio-8810e86354fb4fffde38351f758ebbf722cb2ee1eefc65d72f3123855b0b245a WatchSource:0}: Error finding container 8810e86354fb4fffde38351f758ebbf722cb2ee1eefc65d72f3123855b0b245a: Status 404 returned error can't find the container with id 8810e86354fb4fffde38351f758ebbf722cb2ee1eefc65d72f3123855b0b245a
Jul 28 12:34:47.117232 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:47.117195    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e8e902-7442-4ef7-9545-9a976a21ceb1" path="/var/lib/kubelet/pods/a3e8e902-7442-4ef7-9545-9a976a21ceb1/volumes"
Jul 28 12:34:47.467260 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:47.466769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-w58tj" event={"ID":"14b46c60-e496-4c48-baa3-1780844ca48d","Type":"ContainerStarted","Data":"8810e86354fb4fffde38351f758ebbf722cb2ee1eefc65d72f3123855b0b245a"}
Jul 28 12:34:47.533380 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:47.533336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:47.634529 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:47.634428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:47.733879 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:47.733769    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:47.834618 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:47.834528    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:48.034434 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:48.032907    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:48.094093 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:48.094027    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:48.275543 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:48.275509    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:51.577003 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:51.576974    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:51.654144 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:51.654097    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:51.722988 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:51.722959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:53.606516 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:53.606440    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:53.750422 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:53.750379    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:53.996224 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:53.996131    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:54.081060 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:54.081023    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:54.173378 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:54.173335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:34:54.505770 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:54.505717    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7g7l" event={"ID":"54ff797e-a55b-4020-8e90-97d5bb21db66","Type":"ContainerStarted","Data":"abb5c0fc8d619c405a7c4e023ec4caff0b844908b61671c0be2b611e4fec3e0a"}
Jul 28 12:34:54.508614 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:54.508580    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99" event={"ID":"93fd96f1-c30b-4cbb-b3d7-9385c631443c","Type":"ContainerStarted","Data":"81d5f6b0ed2637ca5bf738d209af9936e7e840c1a242b4a34d7be91a4a354e8e"}
Jul 28 12:34:54.512394 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:54.511946    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7" event={"ID":"75b3692f-d5f5-4012-83b1-9fee03901248","Type":"ContainerStarted","Data":"6c780b8995eeb9eb38fa3159b06353a38925dc83a2e72be37de0b389127bcb77"}
Jul 28 12:34:54.512394 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:54.512264    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:54.513906 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:54.513886    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7"
Jul 28 12:34:54.525667 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:54.525608    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd9d75594-jhr99" podStartSLOduration=1.804426801 podStartE2EDuration="10.525587465s" totalImagesPullingTime="8.721160664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:54.524642073 +0000 UTC m=+81.884246691" watchObservedRunningTime="2026-07-28 12:34:54.525587465 +0000 UTC m=+81.885192071"
Jul 28 12:34:54.546629 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:54.546579    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7db697754b-c6lp7" podStartSLOduration=1.785796279 podStartE2EDuration="10.546558464s" totalImagesPullingTime="8.760762185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:54.546290361 +0000 UTC m=+81.905894967" watchObservedRunningTime="2026-07-28 12:34:54.546558464 +0000 UTC m=+81.906163104"
Jul 28 12:34:56.470779 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:56.470719    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dmbdk"
Jul 28 12:34:58.446294 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:34:58.446250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:00.126889 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.126806    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:00.340809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.340774    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:00.403851 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.403755    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jkbbm"]
Jul 28 12:35:00.411945 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.411911    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.414654 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.414619    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 12:35:00.415030 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.415011    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2zvmv\""
Jul 28 12:35:00.416232 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.416201    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 12:35:00.416343 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.416279    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 12:35:00.416623 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.416403    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 12:35:00.416623 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.416447    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 12:35:00.417405 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.416582    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 12:35:00.437117 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.437079    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:00.510105 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.510068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-wtmp\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.510281 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.510119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-tls\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.510281 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.510147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcr52\" (UniqueName: \"kubernetes.io/projected/a69c6607-ffb6-4416-9e86-9ab57896b3aa-kube-api-access-hcr52\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.510281 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.510182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a69c6607-ffb6-4416-9e86-9ab57896b3aa-metrics-client-ca\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.510281 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.510208    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a69c6607-ffb6-4416-9e86-9ab57896b3aa-sys\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.510281 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.510242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.510543 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.510314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a69c6607-ffb6-4416-9e86-9ab57896b3aa-root\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.510543 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.510384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.510543 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.510441    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-textfile\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.518476 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.518442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:00.591205 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.591168    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:00.611315 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.611238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-wtmp\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.611491 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.611467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-wtmp\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.611756 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.611560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-tls\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.611756 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.611601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcr52\" (UniqueName: \"kubernetes.io/projected/a69c6607-ffb6-4416-9e86-9ab57896b3aa-kube-api-access-hcr52\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.611756 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.611643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a69c6607-ffb6-4416-9e86-9ab57896b3aa-metrics-client-ca\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.611756 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.611672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a69c6607-ffb6-4416-9e86-9ab57896b3aa-sys\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.611756 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.611706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.613014 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.612100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a69c6607-ffb6-4416-9e86-9ab57896b3aa-sys\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.613014 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.612564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a69c6607-ffb6-4416-9e86-9ab57896b3aa-metrics-client-ca\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.613014 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.612702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a69c6607-ffb6-4416-9e86-9ab57896b3aa-root\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.613014 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.612970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.613708 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.613286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a69c6607-ffb6-4416-9e86-9ab57896b3aa-root\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.613708 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.613345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.613708 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.613410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-textfile\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.613708 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.613692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-textfile\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.615273 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.614778    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-tls\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.615797 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.615774    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a69c6607-ffb6-4416-9e86-9ab57896b3aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.621928 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.621846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcr52\" (UniqueName: \"kubernetes.io/projected/a69c6607-ffb6-4416-9e86-9ab57896b3aa-kube-api-access-hcr52\") pod \"node-exporter-jkbbm\" (UID: \"a69c6607-ffb6-4416-9e86-9ab57896b3aa\") " pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.690884 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.690807    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:00.725007 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.724968    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jkbbm"
Jul 28 12:35:00.792451 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.792413    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:00.888920 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.888880    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:00.990092 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:00.989990    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:01.350835 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:01.350799    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:01.454583 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:01.454551    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:01.551229 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:01.551133    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:01.669252 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:01.669152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:03.340475 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:03.340440    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:03.550355 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:03.550322    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:03.616379 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:03.616298    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:03.786235 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:03.786197    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:03.889581 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:03.889498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:05.140383 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:05.140349    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:05.198746 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:05.198701    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:05.495328 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:05.495232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:05.659981 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:05.659944    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:05.760280 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:05.760161    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:06.341128 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:35:06.341092    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69c6607_ffb6_4416_9e86_9ab57896b3aa.slice/crio-4181f5b09152e84f18824f4b2aa779da38f80ddfee2358a6a45a9761c5239e45 WatchSource:0}: Error finding container 4181f5b09152e84f18824f4b2aa779da38f80ddfee2358a6a45a9761c5239e45: Status 404 returned error can't find the container with id 4181f5b09152e84f18824f4b2aa779da38f80ddfee2358a6a45a9761c5239e45
Jul 28 12:35:06.565963 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.565930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jkbbm" event={"ID":"a69c6607-ffb6-4416-9e86-9ab57896b3aa","Type":"ContainerStarted","Data":"4181f5b09152e84f18824f4b2aa779da38f80ddfee2358a6a45a9761c5239e45"}
Jul 28 12:35:06.567389 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.567366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-w58tj" event={"ID":"14b46c60-e496-4c48-baa3-1780844ca48d","Type":"ContainerStarted","Data":"94708a8e9dfa6703db39134298b6e1688a94ff8f414a5dbaf2113d6c5e6b264c"}
Jul 28 12:35:06.567602 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.567585    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-w58tj"
Jul 28 12:35:06.569033 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.569003    2561 patch_prober.go:28] interesting pod/downloads-7c97dc4555-w58tj container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.19:8080/\": dial tcp 10.133.0.19:8080: connect: connection refused" start-of-body=
Jul 28 12:35:06.569129 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.569044    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-w58tj" podUID="14b46c60-e496-4c48-baa3-1780844ca48d" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.19:8080/\": dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 28 12:35:06.585910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.585863    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-w58tj" podStartSLOduration=0.808693734 podStartE2EDuration="20.585849775s" totalImagesPullingTime="19.777156041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:06.58453683 +0000 UTC m=+93.944141434" watchObservedRunningTime="2026-07-28 12:35:06.585849775 +0000 UTC m=+93.945454406"
Jul 28 12:35:06.589217 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.589120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:06.603654 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.603627    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:35:06.622164 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.622138    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.625167 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.625122    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 12:35:06.625566 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.625545    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 12:35:06.625895 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.625876    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 12:35:06.626006 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.625954    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 12:35:06.626006 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.625983    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:35:06.626118 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626084    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-18gsm5eiipvse\""
Jul 28 12:35:06.626118 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626092    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 12:35:06.626214 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626126    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 12:35:06.626214 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626172    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 12:35:06.626313 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626213    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 12:35:06.626313 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626238    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 12:35:06.626313 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626287    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r58hd\""
Jul 28 12:35:06.626456 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626331    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 12:35:06.626456 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626402    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 12:35:06.626552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626457    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 12:35:06.626552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626476    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 12:35:06.626552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.626493    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 12:35:06.628592 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.628576    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 12:35:06.767360 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767568 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767374    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767568 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767568 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767568 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767498    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767568 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767568 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767542    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-config\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767791    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.767818 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.768093 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ttmv\" (UniqueName: \"kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-kube-api-access-4ttmv\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.768093 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.768093 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.768093 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.768093 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767931    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.768093 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.767965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.866229 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.866145    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:06.869249 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869258    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869338 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869505 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-config\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869505 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869505 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869505 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869505 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869505 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869946 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869946 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ttmv\" (UniqueName: \"kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-kube-api-access-4ttmv\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869946 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869946 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869946 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.869946 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.869670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.870174 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.870152    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.870209 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.870195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.870241 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.870221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.870276 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.870240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.870794 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.870770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.870794 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.870787    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.871075 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.870152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.871917 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.871892    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.873701 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.873681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.874847 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.874102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.874847 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.874505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.874847 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.870154    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.875060 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.875034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.875652 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.875627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.875846 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.875822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-config\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.876154 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.876132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.876859 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.876819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.877041 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.877016    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.877941 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.877898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.878036 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.877995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.879060 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.879003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.879218 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.879199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.879761 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.879705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.885261 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.885214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ttmv\" (UniqueName: \"kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-kube-api-access-4ttmv\") pod \"prometheus-k8s-0\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.932910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.932847    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:06.976410 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:06.976355    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:07.135508 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:07.135432    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:35:07.141550 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:35:07.141502    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8644242_ad1e_403d_9aee_918a220c63ee.slice/crio-59bfa1064909c295abd36f61a19c023274ea1205e906a35c9fff98b570747719 WatchSource:0}: Error finding container 59bfa1064909c295abd36f61a19c023274ea1205e906a35c9fff98b570747719: Status 404 returned error can't find the container with id 59bfa1064909c295abd36f61a19c023274ea1205e906a35c9fff98b570747719
Jul 28 12:35:07.574110 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:07.574059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerStarted","Data":"59bfa1064909c295abd36f61a19c023274ea1205e906a35c9fff98b570747719"}
Jul 28 12:35:07.577951 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:07.577896    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7g7l" event={"ID":"54ff797e-a55b-4020-8e90-97d5bb21db66","Type":"ContainerStarted","Data":"30ec9b413cfb428602f92fbc3c4152709993d60c1cd1ba8fdcd20ec2ce42e5fc"}
Jul 28 12:35:07.589824 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:07.589793    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-w58tj"
Jul 28 12:35:08.256817 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:08.256778    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:08.357003 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:08.356968    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:08.599363 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:08.599323    2561 generic.go:356] "Generic (PLEG): container finished" podID="a69c6607-ffb6-4416-9e86-9ab57896b3aa" containerID="58793d478d4c5f33bbc5c6f979682db0194f31ec90a5864ace88b7ad1ebf8f62" exitCode=0
Jul 28 12:35:08.599839 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:08.599473    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jkbbm" event={"ID":"a69c6607-ffb6-4416-9e86-9ab57896b3aa","Type":"ContainerDied","Data":"58793d478d4c5f33bbc5c6f979682db0194f31ec90a5864ace88b7ad1ebf8f62"}
Jul 28 12:35:09.608676 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:09.607217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jkbbm" event={"ID":"a69c6607-ffb6-4416-9e86-9ab57896b3aa","Type":"ContainerStarted","Data":"007e6e9de0b66ddf6ba3d50c48fd4a3af3134a969b0c9889601a9030455fcc01"}
Jul 28 12:35:09.608676 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:09.607347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jkbbm" event={"ID":"a69c6607-ffb6-4416-9e86-9ab57896b3aa","Type":"ContainerStarted","Data":"f55792495c40e5c17b1aa8e9a7f6bf646a44a434d9d2748629deb98cae95c326"}
Jul 28 12:35:09.611927 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:09.610652    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerStarted","Data":"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"}
Jul 28 12:35:09.627776 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:09.627687    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jkbbm" podStartSLOduration=7.661942411 podStartE2EDuration="9.627664216s" totalImagesPullingTime="1.831071946s" totalInitContainerRuntime="134.649859ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:09.625647029 +0000 UTC m=+96.985251635" watchObservedRunningTime="2026-07-28 12:35:09.627664216 +0000 UTC m=+96.987268822"
Jul 28 12:35:09.652989 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:09.652957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:09.754482 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:09.754332    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:10.623511 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:10.623432    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8644242-ad1e-403d-9aee-918a220c63ee" containerID="745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198" exitCode=0
Jul 28 12:35:10.623998 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:10.623527    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerDied","Data":"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"}
Jul 28 12:35:12.160837 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:12.160777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:12.263604 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:12.263570    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:12.434987 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:12.434906    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:12.500603 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:12.500562    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:12.643771 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:12.643716    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:12.745110 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:12.745024    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:13.645275 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:13.645224    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7g7l" event={"ID":"54ff797e-a55b-4020-8e90-97d5bb21db66","Type":"ContainerStarted","Data":"b27320d9cfbc22848860119314238d2e65c9ee141d132b438dd5187824c2e933"}
Jul 28 12:35:13.667066 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:13.667000    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b7g7l" podStartSLOduration=33.608732001999996 podStartE2EDuration="1m4.666977315s" totalImagesPullingTime="31.058245313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:13.665537893 +0000 UTC m=+101.025142499" watchObservedRunningTime="2026-07-28 12:35:13.666977315 +0000 UTC m=+101.026581920"
Jul 28 12:35:14.024578 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:14.024488    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:14.142548 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:14.142510    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:14.226219 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:14.226144    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:16.665978 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:16.665939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerStarted","Data":"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"}
Jul 28 12:35:16.702309 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:16.702278    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:17.469970 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:17.469941    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zn6gv"
Jul 28 12:35:17.672355 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:17.672319    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerStarted","Data":"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"}
Jul 28 12:35:17.799321 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:17.799285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:20.701504 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:20.701456    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:20.803074 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:20.803030    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:21.692238 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:21.692198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerStarted","Data":"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"}
Jul 28 12:35:21.692461 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:21.692282    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerStarted","Data":"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"}
Jul 28 12:35:21.692461 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:21.692323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerStarted","Data":"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"}
Jul 28 12:35:21.692461 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:21.692360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerStarted","Data":"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"}
Jul 28 12:35:21.729682 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:21.729612    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.373239369 podStartE2EDuration="15.72959578s" totalImagesPullingTime="12.334965688s" totalInitContainerRuntime="21.390723ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:21.727836626 +0000 UTC m=+109.087441231" watchObservedRunningTime="2026-07-28 12:35:21.72959578 +0000 UTC m=+109.089200384"
Jul 28 12:35:21.933280 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:21.933238    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:35:22.017497 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:22.017344    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:22.125953 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:22.125908    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:23.670875 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:23.670843    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:23.770348 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:23.770316    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:25.847975 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:25.847936    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:25.912255 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:25.912224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:29.724025 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:29.723992    2561 generic.go:356] "Generic (PLEG): container finished" podID="56dbca26-1e1f-47ec-aa9c-88881434fa9e" containerID="fdd37d5d49e5dc91a9212dc6815227123503de18bb23e511cda33f6a751ee3b8" exitCode=0
Jul 28 12:35:29.724485 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:29.724060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf" event={"ID":"56dbca26-1e1f-47ec-aa9c-88881434fa9e","Type":"ContainerDied","Data":"fdd37d5d49e5dc91a9212dc6815227123503de18bb23e511cda33f6a751ee3b8"}
Jul 28 12:35:29.724485 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:29.724367    2561 scope.go:117] "RemoveContainer" containerID="fdd37d5d49e5dc91a9212dc6815227123503de18bb23e511cda33f6a751ee3b8"
Jul 28 12:35:30.729645 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:30.729609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf" event={"ID":"56dbca26-1e1f-47ec-aa9c-88881434fa9e","Type":"ContainerStarted","Data":"03a242ffe8acd1bb48dc56be1df64c75d87c1301a3bffcfeab9f50de7c33933c"}
Jul 28 12:35:31.744988 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:31.744953    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:31.845652 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:31.845604    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:35.560877 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:35.560829    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:35.622316 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:35.622285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:35.749348 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:35.749313    2561 generic.go:356] "Generic (PLEG): container finished" podID="e19d70c4-98f0-4e28-9ced-f733a41d1dc4" containerID="3caf1e6cac16ba78f08d20edfeec559042a0b872c9bb355c964dbfe90a45358f" exitCode=0
Jul 28 12:35:35.749533 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:35.749375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6" event={"ID":"e19d70c4-98f0-4e28-9ced-f733a41d1dc4","Type":"ContainerDied","Data":"3caf1e6cac16ba78f08d20edfeec559042a0b872c9bb355c964dbfe90a45358f"}
Jul 28 12:35:35.749761 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:35.749721    2561 scope.go:117] "RemoveContainer" containerID="3caf1e6cac16ba78f08d20edfeec559042a0b872c9bb355c964dbfe90a45358f"
Jul 28 12:35:36.754501 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:35:36.754459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p7sz6" event={"ID":"e19d70c4-98f0-4e28-9ced-f733a41d1dc4","Type":"ContainerStarted","Data":"6412a0bb2691ea51a625a5a7415a795d546d2c31d0cf41e5af68909e511126ff"}
Jul 28 12:36:06.933675 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:06.933637    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:06.953203 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:06.953179    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:07.898256 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:07.898230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:19.625088 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:19.625053    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:36:19.829486 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:19.829448    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:36:25.014277 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.014190    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:36:25.014724 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.014587    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="prometheus" containerID="cri-o://5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d" gracePeriod=600
Jul 28 12:36:25.014724 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.014630    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy-thanos" containerID="cri-o://88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f" gracePeriod=600
Jul 28 12:36:25.014724 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.014629    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="thanos-sidecar" containerID="cri-o://ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1" gracePeriod=600
Jul 28 12:36:25.014724 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.014652    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy-web" containerID="cri-o://3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9" gracePeriod=600
Jul 28 12:36:25.014724 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.014680    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="config-reloader" containerID="cri-o://429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2" gracePeriod=600
Jul 28 12:36:25.015027 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.014782    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy" containerID="cri-o://d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f" gracePeriod=600
Jul 28 12:36:25.270641 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.270573    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:25.412383 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412355    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-tls-assets\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.412383 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412388    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-tls\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.412623 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412409    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-0\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.412623 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412430    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.412623 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412456    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-db\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.412802 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412620    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.412802 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412669    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-1\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.412802 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412723    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-web-config\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.412802 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-serving-certs-ca-bundle\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.412802 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412802    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-metrics-client-ca\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412833    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-config-out\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412877    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-2\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412912    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-trusted-ca-bundle\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-config\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412959    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-thanos-prometheus-http-client-file\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.412990    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ttmv\" (UniqueName: \"kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-kube-api-access-4ttmv\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.413015    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-grpc-tls\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.413046    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-kubelet-serving-ca-bundle\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.413074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-kube-rbac-proxy\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413114 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.413101    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-metrics-client-certs\") pod \"a8644242-ad1e-403d-9aee-918a220c63ee\" (UID: \"a8644242-ad1e-403d-9aee-918a220c63ee\") "
Jul 28 12:36:25.413580 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.413263    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:25.413580 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.413338    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.413688 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.413633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:25.413688 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.413669    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:25.413831 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.413798    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:36:25.414130 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.414100    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:25.415132 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.415110    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:25.415459 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.415421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:25.415459 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.415439    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:25.415564 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.415548    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:25.416079 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.415845    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:36:25.416079 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.415885    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:25.416467 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.416442    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-kube-api-access-4ttmv" (OuterVolumeSpecName: "kube-api-access-4ttmv") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "kube-api-access-4ttmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:36:25.416986 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.416929    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-config-out" (OuterVolumeSpecName: "config-out") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:36:25.417078 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.417049    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:25.417866 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.417844    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-config" (OuterVolumeSpecName: "config") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:25.417930 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.417867    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:25.418417 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.418387    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:25.418463 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.418398    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:25.418930 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.418910    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:25.427563 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.427542    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-web-config" (OuterVolumeSpecName: "web-config") pod "a8644242-ad1e-403d-9aee-918a220c63ee" (UID: "a8644242-ad1e-403d-9aee-918a220c63ee"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:25.514559 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514530    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514559 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514562    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514573    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-web-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514583    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-metrics-client-ca\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514592    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-config-out\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514601    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514610    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514619    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514627    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514635    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ttmv\" (UniqueName: \"kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-kube-api-access-4ttmv\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514643    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-grpc-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514651    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514659    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-kube-rbac-proxy\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514668    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-metrics-client-certs\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514675    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8644242-ad1e-403d-9aee-918a220c63ee-tls-assets\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514683    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514691    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514699    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8644242-ad1e-403d-9aee-918a220c63ee-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.514755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.514707    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8644242-ad1e-403d-9aee-918a220c63ee-prometheus-k8s-db\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:36:25.947552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947522    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8644242-ad1e-403d-9aee-918a220c63ee" containerID="88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f" exitCode=0
Jul 28 12:36:25.947552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947546    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8644242-ad1e-403d-9aee-918a220c63ee" containerID="d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f" exitCode=0
Jul 28 12:36:25.947552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947553    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8644242-ad1e-403d-9aee-918a220c63ee" containerID="3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9" exitCode=0
Jul 28 12:36:25.947552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947558    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8644242-ad1e-403d-9aee-918a220c63ee" containerID="ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1" exitCode=0
Jul 28 12:36:25.947552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947564    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8644242-ad1e-403d-9aee-918a220c63ee" containerID="429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2" exitCode=0
Jul 28 12:36:25.947859 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947569    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8644242-ad1e-403d-9aee-918a220c63ee" containerID="5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d" exitCode=0
Jul 28 12:36:25.947859 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947601    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:25.947859 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerDied","Data":"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"}
Jul 28 12:36:25.947859 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerDied","Data":"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"}
Jul 28 12:36:25.947859 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerDied","Data":"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"}
Jul 28 12:36:25.947859 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947755    2561 scope.go:117] "RemoveContainer" containerID="88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"
Jul 28 12:36:25.947859 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerDied","Data":"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"}
Jul 28 12:36:25.948128 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947862    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerDied","Data":"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"}
Jul 28 12:36:25.948128 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerDied","Data":"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"}
Jul 28 12:36:25.948128 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.947948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8644242-ad1e-403d-9aee-918a220c63ee","Type":"ContainerDied","Data":"59bfa1064909c295abd36f61a19c023274ea1205e906a35c9fff98b570747719"}
Jul 28 12:36:25.958590 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.958571    2561 scope.go:117] "RemoveContainer" containerID="d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"
Jul 28 12:36:25.965492 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.965477    2561 scope.go:117] "RemoveContainer" containerID="3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"
Jul 28 12:36:25.971831 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.971715    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:36:25.972757 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.972722    2561 scope.go:117] "RemoveContainer" containerID="ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"
Jul 28 12:36:25.976345 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.976324    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:36:25.979620 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.979599    2561 scope.go:117] "RemoveContainer" containerID="429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"
Jul 28 12:36:25.986232 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.986217    2561 scope.go:117] "RemoveContainer" containerID="5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"
Jul 28 12:36:25.993175 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.993157    2561 scope.go:117] "RemoveContainer" containerID="745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"
Jul 28 12:36:25.999768 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:25.999724    2561 scope.go:117] "RemoveContainer" containerID="88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"
Jul 28 12:36:26.000039 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:36:26.000020    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": container with ID starting with 88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f not found: ID does not exist" containerID="88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"
Jul 28 12:36:26.000111 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.000047    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"} err="failed to get container status \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": rpc error: code = NotFound desc = could not find container \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": container with ID starting with 88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f not found: ID does not exist"
Jul 28 12:36:26.000111 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.000079    2561 scope.go:117] "RemoveContainer" containerID="d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"
Jul 28 12:36:26.000288 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:36:26.000268    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": container with ID starting with d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f not found: ID does not exist" containerID="d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"
Jul 28 12:36:26.000346 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.000298    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"} err="failed to get container status \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": rpc error: code = NotFound desc = could not find container \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": container with ID starting with d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f not found: ID does not exist"
Jul 28 12:36:26.000346 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.000324    2561 scope.go:117] "RemoveContainer" containerID="3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"
Jul 28 12:36:26.000532 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:36:26.000518    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": container with ID starting with 3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9 not found: ID does not exist" containerID="3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"
Jul 28 12:36:26.000570 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.000536    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"} err="failed to get container status \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": rpc error: code = NotFound desc = could not find container \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": container with ID starting with 3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9 not found: ID does not exist"
Jul 28 12:36:26.000570 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.000549    2561 scope.go:117] "RemoveContainer" containerID="ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"
Jul 28 12:36:26.000771 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:36:26.000755    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": container with ID starting with ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1 not found: ID does not exist" containerID="ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"
Jul 28 12:36:26.000821 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.000778    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"} err="failed to get container status \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": rpc error: code = NotFound desc = could not find container \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": container with ID starting with ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1 not found: ID does not exist"
Jul 28 12:36:26.000821 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.000792    2561 scope.go:117] "RemoveContainer" containerID="429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"
Jul 28 12:36:26.000994 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:36:26.000977    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": container with ID starting with 429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2 not found: ID does not exist" containerID="429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"
Jul 28 12:36:26.001053 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.001001    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"} err="failed to get container status \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": rpc error: code = NotFound desc = could not find container \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": container with ID starting with 429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2 not found: ID does not exist"
Jul 28 12:36:26.001053 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.001021    2561 scope.go:117] "RemoveContainer" containerID="5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"
Jul 28 12:36:26.001237 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:36:26.001218    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": container with ID starting with 5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d not found: ID does not exist" containerID="5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"
Jul 28 12:36:26.001276 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.001241    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"} err="failed to get container status \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": rpc error: code = NotFound desc = could not find container \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": container with ID starting with 5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d not found: ID does not exist"
Jul 28 12:36:26.001276 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.001254    2561 scope.go:117] "RemoveContainer" containerID="745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"
Jul 28 12:36:26.001433 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:36:26.001417    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": container with ID starting with 745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198 not found: ID does not exist" containerID="745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"
Jul 28 12:36:26.001472 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.001437    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"} err="failed to get container status \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": rpc error: code = NotFound desc = could not find container \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": container with ID starting with 745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198 not found: ID does not exist"
Jul 28 12:36:26.001472 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.001450    2561 scope.go:117] "RemoveContainer" containerID="88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"
Jul 28 12:36:26.001628 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.001611    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"} err="failed to get container status \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": rpc error: code = NotFound desc = could not find container \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": container with ID starting with 88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f not found: ID does not exist"
Jul 28 12:36:26.001693 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.001630    2561 scope.go:117] "RemoveContainer" containerID="d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"
Jul 28 12:36:26.001840 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.001822    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"} err="failed to get container status \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": rpc error: code = NotFound desc = could not find container \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": container with ID starting with d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f not found: ID does not exist"
Jul 28 12:36:26.001889 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.001841    2561 scope.go:117] "RemoveContainer" containerID="3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"
Jul 28 12:36:26.002055 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.002036    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"} err="failed to get container status \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": rpc error: code = NotFound desc = could not find container \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": container with ID starting with 3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9 not found: ID does not exist"
Jul 28 12:36:26.002124 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.002057    2561 scope.go:117] "RemoveContainer" containerID="ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"
Jul 28 12:36:26.002271 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.002255    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"} err="failed to get container status \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": rpc error: code = NotFound desc = could not find container \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": container with ID starting with ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1 not found: ID does not exist"
Jul 28 12:36:26.002316 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.002272    2561 scope.go:117] "RemoveContainer" containerID="429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"
Jul 28 12:36:26.002455 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.002437    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"} err="failed to get container status \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": rpc error: code = NotFound desc = could not find container \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": container with ID starting with 429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2 not found: ID does not exist"
Jul 28 12:36:26.002492 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.002456    2561 scope.go:117] "RemoveContainer" containerID="5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"
Jul 28 12:36:26.002653 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.002637    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"} err="failed to get container status \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": rpc error: code = NotFound desc = could not find container \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": container with ID starting with 5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d not found: ID does not exist"
Jul 28 12:36:26.002719 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.002654    2561 scope.go:117] "RemoveContainer" containerID="745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"
Jul 28 12:36:26.002904 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.002889    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"} err="failed to get container status \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": rpc error: code = NotFound desc = could not find container \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": container with ID starting with 745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198 not found: ID does not exist"
Jul 28 12:36:26.002963 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.002905    2561 scope.go:117] "RemoveContainer" containerID="88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"
Jul 28 12:36:26.003163 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.003140    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"} err="failed to get container status \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": rpc error: code = NotFound desc = could not find container \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": container with ID starting with 88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f not found: ID does not exist"
Jul 28 12:36:26.003264 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.003164    2561 scope.go:117] "RemoveContainer" containerID="d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"
Jul 28 12:36:26.003552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.003530    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"} err="failed to get container status \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": rpc error: code = NotFound desc = could not find container \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": container with ID starting with d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f not found: ID does not exist"
Jul 28 12:36:26.003552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.003554    2561 scope.go:117] "RemoveContainer" containerID="3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"
Jul 28 12:36:26.003845 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.003824    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"} err="failed to get container status \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": rpc error: code = NotFound desc = could not find container \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": container with ID starting with 3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9 not found: ID does not exist"
Jul 28 12:36:26.003924 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.003848    2561 scope.go:117] "RemoveContainer" containerID="ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"
Jul 28 12:36:26.004131 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.004107    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"} err="failed to get container status \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": rpc error: code = NotFound desc = could not find container \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": container with ID starting with ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1 not found: ID does not exist"
Jul 28 12:36:26.004211 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.004164    2561 scope.go:117] "RemoveContainer" containerID="429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"
Jul 28 12:36:26.004426 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.004401    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"} err="failed to get container status \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": rpc error: code = NotFound desc = could not find container \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": container with ID starting with 429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2 not found: ID does not exist"
Jul 28 12:36:26.004496 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.004428    2561 scope.go:117] "RemoveContainer" containerID="5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"
Jul 28 12:36:26.004645 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.004624    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"} err="failed to get container status \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": rpc error: code = NotFound desc = could not find container \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": container with ID starting with 5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d not found: ID does not exist"
Jul 28 12:36:26.004752 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.004646    2561 scope.go:117] "RemoveContainer" containerID="745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"
Jul 28 12:36:26.004939 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.004918    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"} err="failed to get container status \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": rpc error: code = NotFound desc = could not find container \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": container with ID starting with 745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198 not found: ID does not exist"
Jul 28 12:36:26.005003 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.004940    2561 scope.go:117] "RemoveContainer" containerID="88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"
Jul 28 12:36:26.005110 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005094    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"} err="failed to get container status \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": rpc error: code = NotFound desc = could not find container \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": container with ID starting with 88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f not found: ID does not exist"
Jul 28 12:36:26.005110 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005109    2561 scope.go:117] "RemoveContainer" containerID="d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"
Jul 28 12:36:26.005229 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005142    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:36:26.005360 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005343    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"} err="failed to get container status \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": rpc error: code = NotFound desc = could not find container \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": container with ID starting with d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f not found: ID does not exist"
Jul 28 12:36:26.005416 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005360    2561 scope.go:117] "RemoveContainer" containerID="3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"
Jul 28 12:36:26.005488 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005474    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy-web"
Jul 28 12:36:26.005539 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005490    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy-web"
Jul 28 12:36:26.005539 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005503    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="thanos-sidecar"
Jul 28 12:36:26.005539 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005512    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="thanos-sidecar"
Jul 28 12:36:26.005539 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005519    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy-thanos"
Jul 28 12:36:26.005539 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005524    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy-thanos"
Jul 28 12:36:26.005539 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005534    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="config-reloader"
Jul 28 12:36:26.005539 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005539    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="config-reloader"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005551    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="init-config-reloader"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005557    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="init-config-reloader"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005563    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005568    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005565    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"} err="failed to get container status \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": rpc error: code = NotFound desc = could not find container \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": container with ID starting with 3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9 not found: ID does not exist"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005577    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="prometheus"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005586    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="prometheus"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005636    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="prometheus"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005644    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="config-reloader"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005650    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="thanos-sidecar"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005655    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy-web"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005661    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005666    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" containerName="kube-rbac-proxy-thanos"
Jul 28 12:36:26.005755 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005580    2561 scope.go:117] "RemoveContainer" containerID="ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"
Jul 28 12:36:26.006191 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005931    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"} err="failed to get container status \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": rpc error: code = NotFound desc = could not find container \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": container with ID starting with ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1 not found: ID does not exist"
Jul 28 12:36:26.006191 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.005945    2561 scope.go:117] "RemoveContainer" containerID="429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"
Jul 28 12:36:26.006296 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.006182    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"} err="failed to get container status \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": rpc error: code = NotFound desc = could not find container \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": container with ID starting with 429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2 not found: ID does not exist"
Jul 28 12:36:26.006296 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.006203    2561 scope.go:117] "RemoveContainer" containerID="5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"
Jul 28 12:36:26.006436 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.006417    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"} err="failed to get container status \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": rpc error: code = NotFound desc = could not find container \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": container with ID starting with 5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d not found: ID does not exist"
Jul 28 12:36:26.006472 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.006436    2561 scope.go:117] "RemoveContainer" containerID="745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"
Jul 28 12:36:26.006671 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.006650    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"} err="failed to get container status \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": rpc error: code = NotFound desc = could not find container \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": container with ID starting with 745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198 not found: ID does not exist"
Jul 28 12:36:26.006706 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.006673    2561 scope.go:117] "RemoveContainer" containerID="88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"
Jul 28 12:36:26.006942 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.006920    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"} err="failed to get container status \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": rpc error: code = NotFound desc = could not find container \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": container with ID starting with 88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f not found: ID does not exist"
Jul 28 12:36:26.006942 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.006941    2561 scope.go:117] "RemoveContainer" containerID="d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"
Jul 28 12:36:26.007140 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.007121    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"} err="failed to get container status \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": rpc error: code = NotFound desc = could not find container \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": container with ID starting with d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f not found: ID does not exist"
Jul 28 12:36:26.007192 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.007142    2561 scope.go:117] "RemoveContainer" containerID="3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"
Jul 28 12:36:26.007353 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.007339    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"} err="failed to get container status \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": rpc error: code = NotFound desc = could not find container \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": container with ID starting with 3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9 not found: ID does not exist"
Jul 28 12:36:26.007403 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.007354    2561 scope.go:117] "RemoveContainer" containerID="ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"
Jul 28 12:36:26.007589 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.007565    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"} err="failed to get container status \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": rpc error: code = NotFound desc = could not find container \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": container with ID starting with ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1 not found: ID does not exist"
Jul 28 12:36:26.007589 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.007589    2561 scope.go:117] "RemoveContainer" containerID="429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"
Jul 28 12:36:26.007856 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.007835    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"} err="failed to get container status \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": rpc error: code = NotFound desc = could not find container \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": container with ID starting with 429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2 not found: ID does not exist"
Jul 28 12:36:26.007908 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.007858    2561 scope.go:117] "RemoveContainer" containerID="5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"
Jul 28 12:36:26.008080 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.008064    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"} err="failed to get container status \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": rpc error: code = NotFound desc = could not find container \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": container with ID starting with 5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d not found: ID does not exist"
Jul 28 12:36:26.008131 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.008081    2561 scope.go:117] "RemoveContainer" containerID="745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"
Jul 28 12:36:26.008330 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.008310    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"} err="failed to get container status \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": rpc error: code = NotFound desc = could not find container \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": container with ID starting with 745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198 not found: ID does not exist"
Jul 28 12:36:26.008330 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.008329    2561 scope.go:117] "RemoveContainer" containerID="88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"
Jul 28 12:36:26.008545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.008525    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f"} err="failed to get container status \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": rpc error: code = NotFound desc = could not find container \"88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f\": container with ID starting with 88e117c3cd9071b69e7b618ad47fe046759a6bfc79a16213896cf7339ea77c0f not found: ID does not exist"
Jul 28 12:36:26.008610 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.008547    2561 scope.go:117] "RemoveContainer" containerID="d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"
Jul 28 12:36:26.008804 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.008788    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f"} err="failed to get container status \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": rpc error: code = NotFound desc = could not find container \"d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f\": container with ID starting with d59de8c42f03dd858a1ecc5efe3ef3d29433da627ea73829d8678ecd5861498f not found: ID does not exist"
Jul 28 12:36:26.008858 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.008805    2561 scope.go:117] "RemoveContainer" containerID="3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"
Jul 28 12:36:26.009030 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.009014    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9"} err="failed to get container status \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": rpc error: code = NotFound desc = could not find container \"3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9\": container with ID starting with 3a6caf46855aeb847c6ff905c2103b02243bea1bb193aa4adde4d92a725bece9 not found: ID does not exist"
Jul 28 12:36:26.009073 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.009031    2561 scope.go:117] "RemoveContainer" containerID="ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"
Jul 28 12:36:26.009251 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.009224    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1"} err="failed to get container status \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": rpc error: code = NotFound desc = could not find container \"ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1\": container with ID starting with ee772357703a43bce636930b501de3e685371004fef5645099768005e5576ab1 not found: ID does not exist"
Jul 28 12:36:26.009251 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.009242    2561 scope.go:117] "RemoveContainer" containerID="429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"
Jul 28 12:36:26.009493 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.009469    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2"} err="failed to get container status \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": rpc error: code = NotFound desc = could not find container \"429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2\": container with ID starting with 429e9aef0330d0fbf9ca5567d7d74a73a358aefe41bae76c59ef3349d51a62c2 not found: ID does not exist"
Jul 28 12:36:26.009533 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.009493    2561 scope.go:117] "RemoveContainer" containerID="5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"
Jul 28 12:36:26.009698 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.009679    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d"} err="failed to get container status \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": rpc error: code = NotFound desc = could not find container \"5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d\": container with ID starting with 5f050802a884a9e004d7f8555ec7758b995e1eb67568c58f78fecc6ab590d52d not found: ID does not exist"
Jul 28 12:36:26.009760 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.009700    2561 scope.go:117] "RemoveContainer" containerID="745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"
Jul 28 12:36:26.009907 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.009892    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198"} err="failed to get container status \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": rpc error: code = NotFound desc = could not find container \"745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198\": container with ID starting with 745c5fbd7e24787ef461520575ebafc0624472e620ac388b3bea5991effac198 not found: ID does not exist"
Jul 28 12:36:26.011039 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.011025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.013555 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.013525    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 12:36:26.013821 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.013801    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 12:36:26.013904 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.013821    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 12:36:26.013904 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.013829    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 12:36:26.013904 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.013826    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 12:36:26.013904 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.013896    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 12:36:26.014126 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.014075    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 12:36:26.014273 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.014254    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 12:36:26.014426 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.014384    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r58hd\""
Jul 28 12:36:26.014868 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.014571    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 12:36:26.014868 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.014687    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 12:36:26.015025 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.015005    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 12:36:26.015082 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.015034    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-18gsm5eiipvse\""
Jul 28 12:36:26.015155 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.015139    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 12:36:26.015435 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.015420    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 12:36:26.016317 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.016296    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 12:36:26.019267 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.019249    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 12:36:26.024751 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.024709    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:36:26.118870 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.118842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119027 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.118878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a79ea7e-5093-47be-b591-ee72343c0c94-config-out\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119027 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.118902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-config\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119027 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.118950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119121 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119121 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfd5x\" (UniqueName: \"kubernetes.io/projected/6a79ea7e-5093-47be-b591-ee72343c0c94-kube-api-access-sfd5x\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119121 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119213 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119213 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119213 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119213 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119327 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119327 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119327 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119282    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119327 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119327 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119318    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-web-config\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.119477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.119416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a79ea7e-5093-47be-b591-ee72343c0c94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220041 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.219958    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220041 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.219995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220041 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220041 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220288 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220288 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-web-config\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220288 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220231    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a79ea7e-5093-47be-b591-ee72343c0c94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220288 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a79ea7e-5093-47be-b591-ee72343c0c94-config-out\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220335    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-config\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220364    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220400    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220407    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220425    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfd5x\" (UniqueName: \"kubernetes.io/projected/6a79ea7e-5093-47be-b591-ee72343c0c94-kube-api-access-sfd5x\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220837 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220837 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220837 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220837 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220837 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220837 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.220837 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.220682    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.221164 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.221099    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.221440 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.221397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.222201 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.222173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.222845 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.222816    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.223799 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.223459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.223799 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.223479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.223799 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.223680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-config\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.223799 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.223792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.224292 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.224268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a79ea7e-5093-47be-b591-ee72343c0c94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.224292 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.224281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-web-config\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.224441 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.224413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.225399 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.225369    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.225869 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.225846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.226152 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.226131    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.226223 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.226203    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a79ea7e-5093-47be-b591-ee72343c0c94-config-out\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.226787 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.226771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6a79ea7e-5093-47be-b591-ee72343c0c94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.226965 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.226950    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6a79ea7e-5093-47be-b591-ee72343c0c94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.232445 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.232424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfd5x\" (UniqueName: \"kubernetes.io/projected/6a79ea7e-5093-47be-b591-ee72343c0c94-kube-api-access-sfd5x\") pod \"prometheus-k8s-0\" (UID: \"6a79ea7e-5093-47be-b591-ee72343c0c94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.320559 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.320529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:26.462020 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.461987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:36:26.465244 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:36:26.465219    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a79ea7e_5093_47be_b591_ee72343c0c94.slice/crio-022b9eecd7cf1ff0a6eba03b839fa175b512bf7499f8d2961fea0d3387101a0c WatchSource:0}: Error finding container 022b9eecd7cf1ff0a6eba03b839fa175b512bf7499f8d2961fea0d3387101a0c: Status 404 returned error can't find the container with id 022b9eecd7cf1ff0a6eba03b839fa175b512bf7499f8d2961fea0d3387101a0c
Jul 28 12:36:26.952857 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.952826    2561 generic.go:356] "Generic (PLEG): container finished" podID="6a79ea7e-5093-47be-b591-ee72343c0c94" containerID="c02c0278140842337831811722ad5a140ca2ff6a88cccfb5cc94cef954c02577" exitCode=0
Jul 28 12:36:26.953028 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.952907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a79ea7e-5093-47be-b591-ee72343c0c94","Type":"ContainerDied","Data":"c02c0278140842337831811722ad5a140ca2ff6a88cccfb5cc94cef954c02577"}
Jul 28 12:36:26.953028 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:26.952992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a79ea7e-5093-47be-b591-ee72343c0c94","Type":"ContainerStarted","Data":"022b9eecd7cf1ff0a6eba03b839fa175b512bf7499f8d2961fea0d3387101a0c"}
Jul 28 12:36:27.102952 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:27.102918    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8644242-ad1e-403d-9aee-918a220c63ee" path="/var/lib/kubelet/pods/a8644242-ad1e-403d-9aee-918a220c63ee/volumes"
Jul 28 12:36:27.960329 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:27.960292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a79ea7e-5093-47be-b591-ee72343c0c94","Type":"ContainerStarted","Data":"7dc19fd5fb264788bff48208214d8aa4f5b0d424b13fd8de6e4bcfafc1369509"}
Jul 28 12:36:27.960513 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:27.960378    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a79ea7e-5093-47be-b591-ee72343c0c94","Type":"ContainerStarted","Data":"3b85703e248cc54e8a9a56538fe8fec25f410d7f65668e23da895f97936e3c6b"}
Jul 28 12:36:27.960513 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:27.960417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a79ea7e-5093-47be-b591-ee72343c0c94","Type":"ContainerStarted","Data":"4d29fb5a0b82e5a030d3f32ae26aee2805a43b91b07b5649c8032fbb509f55c3"}
Jul 28 12:36:27.960513 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:27.960455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a79ea7e-5093-47be-b591-ee72343c0c94","Type":"ContainerStarted","Data":"47488c1a691a786d576189c199fde08c6c4cb2fbb7c0479e1cf38245c5994eb7"}
Jul 28 12:36:27.960513 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:27.960497    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a79ea7e-5093-47be-b591-ee72343c0c94","Type":"ContainerStarted","Data":"3ced18dd05e6a8659ff37a41ea3f648c3c4045f0568bb65a553e99b0a1752917"}
Jul 28 12:36:27.960692 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:27.960532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6a79ea7e-5093-47be-b591-ee72343c0c94","Type":"ContainerStarted","Data":"00ee0c1e0b4efb35a181d57b7c0d84560743f251a479f387aacc90370125557f"}
Jul 28 12:36:27.988429 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:27.988351    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.970941092 podStartE2EDuration="2.988334834s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.393742ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:27.986818241 +0000 UTC m=+175.346422845" watchObservedRunningTime="2026-07-28 12:36:27.988334834 +0000 UTC m=+175.347939471"
Jul 28 12:36:29.397314 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:29.397278    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:36:29.453568 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:29.453535    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:36:31.321645 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:31.321605    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:36.895030 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:36.894999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:36:40.219438 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:40.219396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:36:40.270929 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:40.270897    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:36:40.329446 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:40.329416    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:36:49.862487 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:36:49.862452    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:09.750255 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:09.750216    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:26.321218 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:26.321169    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:26.337241 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:26.337212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:27.191938 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:27.191913    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:34.673405 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:34.673373    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:34.895545 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:34.895516    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:34.949231 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:34.949164    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:46.222067 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:46.222032    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:46.277483 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:46.277456    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:46.849549 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:46.849517    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:46.906464 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:46.906435    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:50.563476 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:50.563441    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:50.661968 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:50.661935    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:57.546399 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:57.546296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:57.652636 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:57.652599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:37:57.705177 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:37:57.705137    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:38:02.605848 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:38:02.605809    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:38:32.982053 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:38:32.982022    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:38:32.989751 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:38:32.989705    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:38:32.992270 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:38:32.992249    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 12:40:29.706425 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:40:29.706389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:40:31.012166 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:40:31.012117    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:40:53.743309 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:40:53.743270    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:40:53.849111 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:40:53.849081    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:40:54.131335 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:40:54.131300    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:40:54.257800 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:40:54.257769    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:40:54.362691 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:40:54.362659    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:40:56.795613 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:40:56.795536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:17.191179 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:17.191140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:17.292194 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:17.292148    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:23.553251 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:23.553200    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:23.792221 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:23.792183    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:24.481336 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:24.481298    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:24.725400 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:24.725367    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:24.950131 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:24.950097    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:25.118200 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:25.118153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:25.261309 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:25.261223    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:25.482631 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:25.482593    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:25.697971 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:25.697938    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:26.078697 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:26.078665    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:26.220958 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:26.220928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:31.602761 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:31.602701    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:31.706882 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:31.706817    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:42.414879 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:42.414840    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:42.514969 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:42.514939    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:41:42.616898 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:41:42.616870    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:42:43.195578 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:42:43.195545    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:42:43.318537 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:42:43.318497    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:42:43.453839 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:42:43.453725    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:42:43.574879 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:42:43.574843    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:42:44.067892 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:42:44.067840    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:42:44.179167 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:42:44.179139    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:43:33.013430 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:43:33.013400    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:43:33.018209 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:43:33.018180    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:44:02.838153 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:02.838047    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:44:02.938521 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:02.938477    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:44:03.040340 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:03.040306    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:44:03.100308 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:03.100207    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:44:18.794982 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.794944    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2kv82"]
Jul 28 12:44:18.798176 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.798159    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2kv82"
Jul 28 12:44:18.800878 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.800855    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 12:44:18.800878 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.800870    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 12:44:18.801554 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.801541    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 12:44:18.801594 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.801546    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-44t7z\""
Jul 28 12:44:18.807341 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.807313    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2kv82"]
Jul 28 12:44:18.847126 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.847090    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b452d5ec-0282-4dd5-9361-0104fce274e1-model-cache\") pod \"s3-init-2kv82\" (UID: \"b452d5ec-0282-4dd5-9361-0104fce274e1\") " pod="kserve/s3-init-2kv82"
Jul 28 12:44:18.847126 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.847127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdk6j\" (UniqueName: \"kubernetes.io/projected/b452d5ec-0282-4dd5-9361-0104fce274e1-kube-api-access-zdk6j\") pod \"s3-init-2kv82\" (UID: \"b452d5ec-0282-4dd5-9361-0104fce274e1\") " pod="kserve/s3-init-2kv82"
Jul 28 12:44:18.948532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.948499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b452d5ec-0282-4dd5-9361-0104fce274e1-model-cache\") pod \"s3-init-2kv82\" (UID: \"b452d5ec-0282-4dd5-9361-0104fce274e1\") " pod="kserve/s3-init-2kv82"
Jul 28 12:44:18.948532 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.948539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdk6j\" (UniqueName: \"kubernetes.io/projected/b452d5ec-0282-4dd5-9361-0104fce274e1-kube-api-access-zdk6j\") pod \"s3-init-2kv82\" (UID: \"b452d5ec-0282-4dd5-9361-0104fce274e1\") " pod="kserve/s3-init-2kv82"
Jul 28 12:44:18.948919 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.948898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b452d5ec-0282-4dd5-9361-0104fce274e1-model-cache\") pod \"s3-init-2kv82\" (UID: \"b452d5ec-0282-4dd5-9361-0104fce274e1\") " pod="kserve/s3-init-2kv82"
Jul 28 12:44:18.956833 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:18.956803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdk6j\" (UniqueName: \"kubernetes.io/projected/b452d5ec-0282-4dd5-9361-0104fce274e1-kube-api-access-zdk6j\") pod \"s3-init-2kv82\" (UID: \"b452d5ec-0282-4dd5-9361-0104fce274e1\") " pod="kserve/s3-init-2kv82"
Jul 28 12:44:19.108789 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:19.108694    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2kv82"
Jul 28 12:44:19.242177 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:19.242147    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2kv82"]
Jul 28 12:44:19.245306 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:44:19.245276    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb452d5ec_0282_4dd5_9361_0104fce274e1.slice/crio-7fa3176026d89da5e2fbb7bd8d3ad494415e8a6404487f315f1e7c91acc99263 WatchSource:0}: Error finding container 7fa3176026d89da5e2fbb7bd8d3ad494415e8a6404487f315f1e7c91acc99263: Status 404 returned error can't find the container with id 7fa3176026d89da5e2fbb7bd8d3ad494415e8a6404487f315f1e7c91acc99263
Jul 28 12:44:19.247687 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:19.247669    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:44:19.696436 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:19.696401    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2kv82" event={"ID":"b452d5ec-0282-4dd5-9361-0104fce274e1","Type":"ContainerStarted","Data":"7fa3176026d89da5e2fbb7bd8d3ad494415e8a6404487f315f1e7c91acc99263"}
Jul 28 12:44:22.710095 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:22.710061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2kv82" event={"ID":"b452d5ec-0282-4dd5-9361-0104fce274e1","Type":"ContainerStarted","Data":"a81ce5c92a2222ac36768c9fb97d27f225902841eadb679cd352acb520bd7f71"}
Jul 28 12:44:25.723449 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:25.723413    2561 generic.go:356] "Generic (PLEG): container finished" podID="b452d5ec-0282-4dd5-9361-0104fce274e1" containerID="a81ce5c92a2222ac36768c9fb97d27f225902841eadb679cd352acb520bd7f71" exitCode=0
Jul 28 12:44:25.723846 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:25.723483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2kv82" event={"ID":"b452d5ec-0282-4dd5-9361-0104fce274e1","Type":"ContainerDied","Data":"a81ce5c92a2222ac36768c9fb97d27f225902841eadb679cd352acb520bd7f71"}
Jul 28 12:44:30.749010 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:30.748973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2kv82" event={"ID":"b452d5ec-0282-4dd5-9361-0104fce274e1","Type":"ContainerStarted","Data":"0193e427a8654366de95e3ad1d6e0010af589e667951ba2aa2a6f05265c0ea86"}
Jul 28 12:44:30.768521 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:44:30.768450    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-2kv82" podStartSLOduration=2.411645437 podStartE2EDuration="12.768433299s" totalImagesPullingTime="7.767498617s" totalInitContainerRuntime="2.589289245s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:44:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:44:30.766381259 +0000 UTC m=+658.125985866" watchObservedRunningTime="2026-07-28 12:44:30.768433299 +0000 UTC m=+658.128037906"
Jul 28 12:45:35.996576 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:35.996540    2561 generic.go:356] "Generic (PLEG): container finished" podID="b452d5ec-0282-4dd5-9361-0104fce274e1" containerID="0193e427a8654366de95e3ad1d6e0010af589e667951ba2aa2a6f05265c0ea86" exitCode=0
Jul 28 12:45:35.997006 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:35.996609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2kv82" event={"ID":"b452d5ec-0282-4dd5-9361-0104fce274e1","Type":"ContainerDied","Data":"0193e427a8654366de95e3ad1d6e0010af589e667951ba2aa2a6f05265c0ea86"}
Jul 28 12:45:37.125813 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:37.125787    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2kv82"
Jul 28 12:45:37.289001 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:37.288971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b452d5ec-0282-4dd5-9361-0104fce274e1-model-cache\") pod \"b452d5ec-0282-4dd5-9361-0104fce274e1\" (UID: \"b452d5ec-0282-4dd5-9361-0104fce274e1\") "
Jul 28 12:45:37.289154 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:37.289011    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdk6j\" (UniqueName: \"kubernetes.io/projected/b452d5ec-0282-4dd5-9361-0104fce274e1-kube-api-access-zdk6j\") pod \"b452d5ec-0282-4dd5-9361-0104fce274e1\" (UID: \"b452d5ec-0282-4dd5-9361-0104fce274e1\") "
Jul 28 12:45:37.291493 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:37.291466    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b452d5ec-0282-4dd5-9361-0104fce274e1-kube-api-access-zdk6j" (OuterVolumeSpecName: "kube-api-access-zdk6j") pod "b452d5ec-0282-4dd5-9361-0104fce274e1" (UID: "b452d5ec-0282-4dd5-9361-0104fce274e1"). InnerVolumeSpecName "kube-api-access-zdk6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:45:37.308627 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:37.308594    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b452d5ec-0282-4dd5-9361-0104fce274e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b452d5ec-0282-4dd5-9361-0104fce274e1" (UID: "b452d5ec-0282-4dd5-9361-0104fce274e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:45:37.389691 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:37.389656    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b452d5ec-0282-4dd5-9361-0104fce274e1-model-cache\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:45:37.389691 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:37.389684    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdk6j\" (UniqueName: \"kubernetes.io/projected/b452d5ec-0282-4dd5-9361-0104fce274e1-kube-api-access-zdk6j\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:45:38.005379 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:38.005348    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2kv82"
Jul 28 12:45:38.005551 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:38.005384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2kv82" event={"ID":"b452d5ec-0282-4dd5-9361-0104fce274e1","Type":"ContainerDied","Data":"7fa3176026d89da5e2fbb7bd8d3ad494415e8a6404487f315f1e7c91acc99263"}
Jul 28 12:45:38.005551 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:38.005417    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fa3176026d89da5e2fbb7bd8d3ad494415e8a6404487f315f1e7c91acc99263"
Jul 28 12:45:38.416657 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:38.416625    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:45:40.348894 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:40.348847    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:45:51.475538 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.475494    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:45:51.610307 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.610278    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"]
Jul 28 12:45:51.610585 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.610574    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b452d5ec-0282-4dd5-9361-0104fce274e1" containerName="download-hf-model"
Jul 28 12:45:51.610627 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.610588    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b452d5ec-0282-4dd5-9361-0104fce274e1" containerName="download-hf-model"
Jul 28 12:45:51.610627 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.610606    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b452d5ec-0282-4dd5-9361-0104fce274e1" containerName="s3-init"
Jul 28 12:45:51.610627 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.610612    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b452d5ec-0282-4dd5-9361-0104fce274e1" containerName="s3-init"
Jul 28 12:45:51.610716 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.610667    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b452d5ec-0282-4dd5-9361-0104fce274e1" containerName="s3-init"
Jul 28 12:45:51.614170 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.614154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.616989 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.616961    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 12:45:51.617151 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.617129    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-51250-predictor-serving-cert\""
Jul 28 12:45:51.617211 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.617178    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-51250-kube-rbac-proxy-sar-config\""
Jul 28 12:45:51.617268 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.617218    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rtw6d\""
Jul 28 12:45:51.617342 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.617284    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 12:45:51.617980 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.617957    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 12:45:51.618102 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.618002    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 12:45:51.628369 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.628343    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"]
Jul 28 12:45:51.698564 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.698521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.698564 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.698563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-proxy-tls\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.698816 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.698612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.698816 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.698631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlf7\" (UniqueName: \"kubernetes.io/projected/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-api-access-jvlf7\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.698816 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.698668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.799747 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.799704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.799964 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.799771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlf7\" (UniqueName: \"kubernetes.io/projected/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-api-access-jvlf7\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.799964 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.799811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.799964 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.799836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.799964 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.799857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-proxy-tls\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.799964 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:45:51.799963    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-51250-predictor-serving-cert" not found
Jul 28 12:45:51.800222 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:45:51.800040    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-proxy-tls podName:7dbd61fa-0c34-4083-8343-1fc2c4d43a82 nodeName:}" failed. No retries permitted until 2026-07-28 12:45:52.300018156 +0000 UTC m=+739.659622743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-proxy-tls") pod "isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" (UID: "7dbd61fa-0c34-4083-8343-1fc2c4d43a82") : secret "isvc-raw-sklearn-batcher-51250-predictor-serving-cert" not found
Jul 28 12:45:51.800301 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.800280    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.800452 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.800433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.800505 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.800488    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:51.809749 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:51.809705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlf7\" (UniqueName: \"kubernetes.io/projected/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-api-access-jvlf7\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:52.003657 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:52.003620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:45:52.304035 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:52.303991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-proxy-tls\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:52.306623 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:52.306594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-proxy-tls\") pod \"isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:52.524801 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:52.524722    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:45:52.663585 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:52.663562    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"]
Jul 28 12:45:52.666030 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:45:52.666002    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbd61fa_0c34_4083_8343_1fc2c4d43a82.slice/crio-e18e21968de69c6b52950ddfe3312189b71317f3c06864cbcda12dfd2c068043 WatchSource:0}: Error finding container e18e21968de69c6b52950ddfe3312189b71317f3c06864cbcda12dfd2c068043: Status 404 returned error can't find the container with id e18e21968de69c6b52950ddfe3312189b71317f3c06864cbcda12dfd2c068043
Jul 28 12:45:53.065459 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:53.065418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" event={"ID":"7dbd61fa-0c34-4083-8343-1fc2c4d43a82","Type":"ContainerStarted","Data":"63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c"}
Jul 28 12:45:53.065664 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:53.065492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" event={"ID":"7dbd61fa-0c34-4083-8343-1fc2c4d43a82","Type":"ContainerStarted","Data":"e18e21968de69c6b52950ddfe3312189b71317f3c06864cbcda12dfd2c068043"}
Jul 28 12:45:54.661666 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:54.661619    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:45:57.082800 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:57.082764    2561 generic.go:356] "Generic (PLEG): container finished" podID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerID="63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c" exitCode=0
Jul 28 12:45:57.083196 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:45:57.082816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" event={"ID":"7dbd61fa-0c34-4083-8343-1fc2c4d43a82","Type":"ContainerDied","Data":"63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c"}
Jul 28 12:46:10.151896 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:10.151857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" event={"ID":"7dbd61fa-0c34-4083-8343-1fc2c4d43a82","Type":"ContainerStarted","Data":"ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d"}
Jul 28 12:46:13.170136 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:13.170086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" event={"ID":"7dbd61fa-0c34-4083-8343-1fc2c4d43a82","Type":"ContainerStarted","Data":"cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398"}
Jul 28 12:46:15.181709 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:15.181671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" event={"ID":"7dbd61fa-0c34-4083-8343-1fc2c4d43a82","Type":"ContainerStarted","Data":"09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79"}
Jul 28 12:46:15.182183 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:15.181893    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:46:15.182183 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:15.181960    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:46:15.182183 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:15.181996    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:46:15.183369 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:15.183337    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:46:15.184035 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:15.184012    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:15.203158 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:15.203111    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podStartSLOduration=3.5096536499999997 podStartE2EDuration="24.203098523s" totalImagesPullingTime="17.21265415s" totalInitContainerRuntime="3.480790723s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:45:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:46:15.202013206 +0000 UTC m=+762.561617861" watchObservedRunningTime="2026-07-28 12:46:15.203098523 +0000 UTC m=+762.562703128"
Jul 28 12:46:16.186018 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:16.185978    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:46:16.186477 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:16.186372    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:21.190482 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:21.190450    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:46:21.191096 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:21.191067    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:46:21.191344 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:21.191321    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:23.774697 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:23.774657    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:46:23.833504 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:23.833458    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:46:31.191792 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:31.191706    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:46:31.192202 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:31.192076    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:41.190952 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:41.190909    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:46:41.191481 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:41.191276    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:43.985337 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:43.985294    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:46:51.191872 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:51.191824    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:46:51.192300 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:46:51.192275    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:01.191821 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:01.191708    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:47:01.192230 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:01.192119    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:04.726214 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:04.726178    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:47:11.191377 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:11.191330    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:47:11.191801 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:11.191770    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:21.191862 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:21.191818    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:47:21.192382 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:21.192362    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:31.191835 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:31.191804    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:47:31.192287 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:31.192050    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:47:46.809332 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:46.809295    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"]
Jul 28 12:47:46.809764 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:46.809622    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" containerID="cri-o://ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d" gracePeriod=30
Jul 28 12:47:46.809764 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:46.809705    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kube-rbac-proxy" containerID="cri-o://cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398" gracePeriod=30
Jul 28 12:47:46.809926 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:46.809863    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" containerID="cri-o://09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79" gracePeriod=30
Jul 28 12:47:46.860398 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:46.860368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:47:46.976509 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:46.976476    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:47:46.983841 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:46.983813    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"]
Jul 28 12:47:46.987595 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:46.987573    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:46.992394 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:46.992374    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e5cf3-kube-rbac-proxy-sar-config\""
Jul 28 12:47:46.992515 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:46.992413    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-e5cf3-predictor-serving-cert\""
Jul 28 12:47:47.011203 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.011166    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"]
Jul 28 12:47:47.130529 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.130442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:47:47.135466 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.135443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10baf772-a265-4dbb-8afa-29fe5877f2e4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.135563 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.135506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzcr\" (UniqueName: \"kubernetes.io/projected/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-api-access-9kzcr\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.135563 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.135535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10baf772-a265-4dbb-8afa-29fe5877f2e4-proxy-tls\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.135563 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.135552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.135765 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.135627    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.236536 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.236506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10baf772-a265-4dbb-8afa-29fe5877f2e4-proxy-tls\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.236536 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.236540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.236784 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.236566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.236784 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.236625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10baf772-a265-4dbb-8afa-29fe5877f2e4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.236903 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.236841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzcr\" (UniqueName: \"kubernetes.io/projected/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-api-access-9kzcr\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.237198 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.237172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10baf772-a265-4dbb-8afa-29fe5877f2e4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.237460 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.237443    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.237522 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.237485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.239029 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.239012    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10baf772-a265-4dbb-8afa-29fe5877f2e4-proxy-tls\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.261958 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.261938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzcr\" (UniqueName: \"kubernetes.io/projected/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-api-access-9kzcr\") pod \"isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.300810 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.300784    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:47.465909 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.465859    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"]
Jul 28 12:47:47.468163 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:47:47.468133    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10baf772_a265_4dbb_8afa_29fe5877f2e4.slice/crio-7559c206f853b792072cada7b6509d25710a55e3ec098525203c34807524a6a9 WatchSource:0}: Error finding container 7559c206f853b792072cada7b6509d25710a55e3ec098525203c34807524a6a9: Status 404 returned error can't find the container with id 7559c206f853b792072cada7b6509d25710a55e3ec098525203c34807524a6a9
Jul 28 12:47:47.533988 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.533961    2561 generic.go:356] "Generic (PLEG): container finished" podID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerID="cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398" exitCode=2
Jul 28 12:47:47.534084 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.534033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" event={"ID":"7dbd61fa-0c34-4083-8343-1fc2c4d43a82","Type":"ContainerDied","Data":"cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398"}
Jul 28 12:47:47.535614 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.535595    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" event={"ID":"10baf772-a265-4dbb-8afa-29fe5877f2e4","Type":"ContainerStarted","Data":"7559c206f853b792072cada7b6509d25710a55e3ec098525203c34807524a6a9"}
Jul 28 12:47:47.712686 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:47.712596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:47:48.541248 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:48.541211    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" event={"ID":"10baf772-a265-4dbb-8afa-29fe5877f2e4","Type":"ContainerStarted","Data":"dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8"}
Jul 28 12:47:51.186557 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:51.186514    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 28 12:47:51.191320 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:51.191274    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:47:51.191638 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:51.191612    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:51.555214 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:51.555187    2561 generic.go:356] "Generic (PLEG): container finished" podID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerID="ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d" exitCode=0
Jul 28 12:47:51.555361 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:51.555260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" event={"ID":"7dbd61fa-0c34-4083-8343-1fc2c4d43a82","Type":"ContainerDied","Data":"ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d"}
Jul 28 12:47:51.556832 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:51.556812    2561 generic.go:356] "Generic (PLEG): container finished" podID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerID="dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8" exitCode=0
Jul 28 12:47:51.556926 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:51.556852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" event={"ID":"10baf772-a265-4dbb-8afa-29fe5877f2e4","Type":"ContainerDied","Data":"dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8"}
Jul 28 12:47:52.562424 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:52.562387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" event={"ID":"10baf772-a265-4dbb-8afa-29fe5877f2e4","Type":"ContainerStarted","Data":"9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65"}
Jul 28 12:47:52.562800 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:52.562458    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" event={"ID":"10baf772-a265-4dbb-8afa-29fe5877f2e4","Type":"ContainerStarted","Data":"02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31"}
Jul 28 12:47:52.562800 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:52.562759    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:52.593174 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:52.593128    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podStartSLOduration=2.915497644 podStartE2EDuration="6.593113551s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.677615907s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:47:52.59054997 +0000 UTC m=+859.950154575" watchObservedRunningTime="2026-07-28 12:47:52.593113551 +0000 UTC m=+859.952718148"
Jul 28 12:47:53.566102 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:53.566066    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:53.567430 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:53.567400    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 12:47:54.569563 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:54.569527    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 12:47:56.186484 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:56.186437    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 28 12:47:59.574703 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:59.574676    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:47:59.575173 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:47:59.575148    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 12:48:01.186289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:01.186246    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 28 12:48:01.186757 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:01.186372    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:48:01.191541 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:01.191501    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:48:01.191854 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:01.191830    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:06.186937 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:06.186896    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 28 12:48:09.575664 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:09.575621    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 12:48:11.186448 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:11.186411    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 28 12:48:11.191902 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:11.191867    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 12:48:11.192040 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:11.191989    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:48:11.192198 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:11.192176    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:11.192284 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:11.192272    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:48:16.186188 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:16.186150    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 28 12:48:16.967171 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:16.967147    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:48:17.060045 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.060015    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kserve-provision-location\") pod \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") "
Jul 28 12:48:17.060201 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.060066    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-proxy-tls\") pod \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") "
Jul 28 12:48:17.060201 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.060084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-cabundle-cert\") pod \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") "
Jul 28 12:48:17.060302 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.060278    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-rbac-proxy-sar-config\") pod \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") "
Jul 28 12:48:17.060348 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.060305    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7dbd61fa-0c34-4083-8343-1fc2c4d43a82" (UID: "7dbd61fa-0c34-4083-8343-1fc2c4d43a82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:48:17.060348 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.060341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvlf7\" (UniqueName: \"kubernetes.io/projected/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-api-access-jvlf7\") pod \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\" (UID: \"7dbd61fa-0c34-4083-8343-1fc2c4d43a82\") "
Jul 28 12:48:17.060925 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.060366    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7dbd61fa-0c34-4083-8343-1fc2c4d43a82" (UID: "7dbd61fa-0c34-4083-8343-1fc2c4d43a82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:48:17.060925 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.060545    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:48:17.060925 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.060557    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:48:17.060925 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.060567    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7dbd61fa-0c34-4083-8343-1fc2c4d43a82" (UID: "7dbd61fa-0c34-4083-8343-1fc2c4d43a82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:48:17.062401 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.062377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7dbd61fa-0c34-4083-8343-1fc2c4d43a82" (UID: "7dbd61fa-0c34-4083-8343-1fc2c4d43a82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:48:17.062505 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.062413    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-api-access-jvlf7" (OuterVolumeSpecName: "kube-api-access-jvlf7") pod "7dbd61fa-0c34-4083-8343-1fc2c4d43a82" (UID: "7dbd61fa-0c34-4083-8343-1fc2c4d43a82"). InnerVolumeSpecName "kube-api-access-jvlf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:48:17.161237 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.161194    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:48:17.161237 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.161224    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:48:17.161237 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.161239    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvlf7\" (UniqueName: \"kubernetes.io/projected/7dbd61fa-0c34-4083-8343-1fc2c4d43a82-kube-api-access-jvlf7\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:48:17.670059 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.670022    2561 generic.go:356] "Generic (PLEG): container finished" podID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerID="09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79" exitCode=0
Jul 28 12:48:17.670524 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.670089    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"
Jul 28 12:48:17.670524 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.670097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" event={"ID":"7dbd61fa-0c34-4083-8343-1fc2c4d43a82","Type":"ContainerDied","Data":"09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79"}
Jul 28 12:48:17.670524 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.670161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt" event={"ID":"7dbd61fa-0c34-4083-8343-1fc2c4d43a82","Type":"ContainerDied","Data":"e18e21968de69c6b52950ddfe3312189b71317f3c06864cbcda12dfd2c068043"}
Jul 28 12:48:17.670524 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.670191    2561 scope.go:117] "RemoveContainer" containerID="09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79"
Jul 28 12:48:17.693539 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.693519    2561 scope.go:117] "RemoveContainer" containerID="cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398"
Jul 28 12:48:17.701197 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.701177    2561 scope.go:117] "RemoveContainer" containerID="ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d"
Jul 28 12:48:17.705411 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.705387    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"]
Jul 28 12:48:17.710951 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.710929    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-51250-predictor-99c5fcf56-8nzjt"]
Jul 28 12:48:17.712801 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.712779    2561 scope.go:117] "RemoveContainer" containerID="63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c"
Jul 28 12:48:17.720793 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.720774    2561 scope.go:117] "RemoveContainer" containerID="09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79"
Jul 28 12:48:17.721080 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:48:17.721062    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79\": container with ID starting with 09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79 not found: ID does not exist" containerID="09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79"
Jul 28 12:48:17.721155 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.721088    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79"} err="failed to get container status \"09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79\": rpc error: code = NotFound desc = could not find container \"09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79\": container with ID starting with 09af4d026aced10b28733d0f76b147591ef92d82c663aa3f323567aaf5fcbb79 not found: ID does not exist"
Jul 28 12:48:17.721155 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.721109    2561 scope.go:117] "RemoveContainer" containerID="cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398"
Jul 28 12:48:17.721304 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:48:17.721289    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398\": container with ID starting with cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398 not found: ID does not exist" containerID="cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398"
Jul 28 12:48:17.721365 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.721305    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398"} err="failed to get container status \"cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398\": rpc error: code = NotFound desc = could not find container \"cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398\": container with ID starting with cf0952ed698f23b80ea271c9d531cfbf1bdba886005a0640053888f30d3a6398 not found: ID does not exist"
Jul 28 12:48:17.721365 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.721318    2561 scope.go:117] "RemoveContainer" containerID="ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d"
Jul 28 12:48:17.721532 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:48:17.721513    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d\": container with ID starting with ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d not found: ID does not exist" containerID="ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d"
Jul 28 12:48:17.721615 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.721538    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d"} err="failed to get container status \"ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d\": rpc error: code = NotFound desc = could not find container \"ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d\": container with ID starting with ea2106278bda5ad6c6974590464185807f3b8e5bac80308554de12be5c8a4e9d not found: ID does not exist"
Jul 28 12:48:17.721615 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.721556    2561 scope.go:117] "RemoveContainer" containerID="63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c"
Jul 28 12:48:17.721841 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:48:17.721822    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c\": container with ID starting with 63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c not found: ID does not exist" containerID="63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c"
Jul 28 12:48:17.721910 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:17.721842    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c"} err="failed to get container status \"63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c\": rpc error: code = NotFound desc = could not find container \"63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c\": container with ID starting with 63d984a6eaf57e8bb30d7c141a825e71813e183b8240e6f1fce1d8d9a6e2445c not found: ID does not exist"
Jul 28 12:48:19.101947 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:19.101912    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" path="/var/lib/kubelet/pods/7dbd61fa-0c34-4083-8343-1fc2c4d43a82/volumes"
Jul 28 12:48:19.575717 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:19.575676    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 12:48:29.575638 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:29.575560    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 12:48:33.039196 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:33.039169    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:48:33.041691 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:33.041655    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:48:39.576145 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:39.576104    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 12:48:49.578643 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:49.578603    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 12:48:59.575568 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:48:59.575526    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 12:49:09.576465 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:09.576434    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:49:26.998012 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:26.997954    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:49:27.080871 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:27.080828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:49:37.033671 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.033630    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:49:37.174298 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.174267    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"]
Jul 28 12:49:37.174544 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.174492    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" containerID="cri-o://02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31" gracePeriod=30
Jul 28 12:49:37.174544 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.174526    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kube-rbac-proxy" containerID="cri-o://9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65" gracePeriod=30
Jul 28 12:49:37.192297 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.192270    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:49:37.295212 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.295186    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:49:37.350934 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.350901    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"]
Jul 28 12:49:37.351388 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351368    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container"
Jul 28 12:49:37.351471 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351392    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container"
Jul 28 12:49:37.351471 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351419    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="storage-initializer"
Jul 28 12:49:37.351471 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351429    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="storage-initializer"
Jul 28 12:49:37.351471 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351443    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent"
Jul 28 12:49:37.351471 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351451    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent"
Jul 28 12:49:37.351804 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351472    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kube-rbac-proxy"
Jul 28 12:49:37.351804 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351482    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kube-rbac-proxy"
Jul 28 12:49:37.351804 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351599    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="agent"
Jul 28 12:49:37.351804 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351612    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kube-rbac-proxy"
Jul 28 12:49:37.351804 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.351621    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dbd61fa-0c34-4083-8343-1fc2c4d43a82" containerName="kserve-container"
Jul 28 12:49:37.355187 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.355161    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.358208 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.358186    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-7f8c4-kube-rbac-proxy-sar-config\""
Jul 28 12:49:37.358334 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.358295    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-serving-cert\""
Jul 28 12:49:37.370373 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.370347    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"]
Jul 28 12:49:37.399536 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.399502    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:49:37.490404 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.490372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.490571 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.490430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-542g2\" (UniqueName: \"kubernetes.io/projected/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-api-access-542g2\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.490571 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.490519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.490571 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.490561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.490779 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.490624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.491409 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.491385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:49:37.591725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.591649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-542g2\" (UniqueName: \"kubernetes.io/projected/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-api-access-542g2\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.591725 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.591705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.591929 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.591765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.591929 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.591809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.591929 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.591844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.591929 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:49:37.591860    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-7f8c4-predictor-serving-cert" not found
Jul 28 12:49:37.592057 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:49:37.591938    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-proxy-tls podName:fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2 nodeName:}" failed. No retries permitted until 2026-07-28 12:49:38.091917465 +0000 UTC m=+965.451522050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" (UID: "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2") : secret "isvc-sklearn-graph-raw-hpa-7f8c4-predictor-serving-cert" not found
Jul 28 12:49:37.592293 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.592270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.592622 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.592604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.592664 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.592639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.602435 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.602411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-542g2\" (UniqueName: \"kubernetes.io/projected/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-api-access-542g2\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:37.700341 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.700300    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:49:37.976775 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.976675    2561 generic.go:356] "Generic (PLEG): container finished" podID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerID="9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65" exitCode=2
Jul 28 12:49:37.976775 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:37.976724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" event={"ID":"10baf772-a265-4dbb-8afa-29fe5877f2e4","Type":"ContainerDied","Data":"9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65"}
Jul 28 12:49:38.085721 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:38.085685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:49:38.096797 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:38.096770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:38.099312 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:38.099277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:38.265472 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:38.265380    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:38.397421 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:38.397398    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"]
Jul 28 12:49:38.399984 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:49:38.399942    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5354fb_a5a6_4e4f_b531_c31d6aadd0e2.slice/crio-cc14ff15607a6d6f0485eded24ce3d1ff451dda79b959d1bce4ce535a34646ba WatchSource:0}: Error finding container cc14ff15607a6d6f0485eded24ce3d1ff451dda79b959d1bce4ce535a34646ba: Status 404 returned error can't find the container with id cc14ff15607a6d6f0485eded24ce3d1ff451dda79b959d1bce4ce535a34646ba
Jul 28 12:49:38.982478 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:38.982444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" event={"ID":"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2","Type":"ContainerStarted","Data":"14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95"}
Jul 28 12:49:38.982654 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:38.982505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" event={"ID":"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2","Type":"ContainerStarted","Data":"cc14ff15607a6d6f0485eded24ce3d1ff451dda79b959d1bce4ce535a34646ba"}
Jul 28 12:49:39.570658 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:39.570618    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 28 12:49:39.575944 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:39.575923    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 12:49:41.819304 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.819270    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:49:41.929095 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.928983    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10baf772-a265-4dbb-8afa-29fe5877f2e4-proxy-tls\") pod \"10baf772-a265-4dbb-8afa-29fe5877f2e4\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") "
Jul 28 12:49:41.929095 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.929046    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-rbac-proxy-sar-config\") pod \"10baf772-a265-4dbb-8afa-29fe5877f2e4\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") "
Jul 28 12:49:41.929095 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.929098    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10baf772-a265-4dbb-8afa-29fe5877f2e4-kserve-provision-location\") pod \"10baf772-a265-4dbb-8afa-29fe5877f2e4\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") "
Jul 28 12:49:41.929362 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.929137    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-cabundle-cert\") pod \"10baf772-a265-4dbb-8afa-29fe5877f2e4\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") "
Jul 28 12:49:41.929362 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.929161    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kzcr\" (UniqueName: \"kubernetes.io/projected/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-api-access-9kzcr\") pod \"10baf772-a265-4dbb-8afa-29fe5877f2e4\" (UID: \"10baf772-a265-4dbb-8afa-29fe5877f2e4\") "
Jul 28 12:49:41.929443 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.929421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "10baf772-a265-4dbb-8afa-29fe5877f2e4" (UID: "10baf772-a265-4dbb-8afa-29fe5877f2e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:49:41.929478 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.929421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10baf772-a265-4dbb-8afa-29fe5877f2e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10baf772-a265-4dbb-8afa-29fe5877f2e4" (UID: "10baf772-a265-4dbb-8afa-29fe5877f2e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:49:41.929515 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.929481    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "10baf772-a265-4dbb-8afa-29fe5877f2e4" (UID: "10baf772-a265-4dbb-8afa-29fe5877f2e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:49:41.931339 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.931306    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-api-access-9kzcr" (OuterVolumeSpecName: "kube-api-access-9kzcr") pod "10baf772-a265-4dbb-8afa-29fe5877f2e4" (UID: "10baf772-a265-4dbb-8afa-29fe5877f2e4"). InnerVolumeSpecName "kube-api-access-9kzcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:49:41.931453 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.931363    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10baf772-a265-4dbb-8afa-29fe5877f2e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10baf772-a265-4dbb-8afa-29fe5877f2e4" (UID: "10baf772-a265-4dbb-8afa-29fe5877f2e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:49:41.996247 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.996211    2561 generic.go:356] "Generic (PLEG): container finished" podID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerID="02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31" exitCode=0
Jul 28 12:49:41.996396 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.996272    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"
Jul 28 12:49:41.996396 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.996292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" event={"ID":"10baf772-a265-4dbb-8afa-29fe5877f2e4","Type":"ContainerDied","Data":"02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31"}
Jul 28 12:49:41.996396 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.996356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx" event={"ID":"10baf772-a265-4dbb-8afa-29fe5877f2e4","Type":"ContainerDied","Data":"7559c206f853b792072cada7b6509d25710a55e3ec098525203c34807524a6a9"}
Jul 28 12:49:41.996396 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:41.996382    2561 scope.go:117] "RemoveContainer" containerID="9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65"
Jul 28 12:49:42.004722 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.004704    2561 scope.go:117] "RemoveContainer" containerID="02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31"
Jul 28 12:49:42.012267 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.012239    2561 scope.go:117] "RemoveContainer" containerID="dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8"
Jul 28 12:49:42.019449 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.019431    2561 scope.go:117] "RemoveContainer" containerID="9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65"
Jul 28 12:49:42.019833 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:49:42.019718    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65\": container with ID starting with 9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65 not found: ID does not exist" containerID="9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65"
Jul 28 12:49:42.019833 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.019793    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65"} err="failed to get container status \"9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65\": rpc error: code = NotFound desc = could not find container \"9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65\": container with ID starting with 9abbdbb9d98002890fd5f5830a76c2836718cdc79e720961f83c36f4bfcceb65 not found: ID does not exist"
Jul 28 12:49:42.019833 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.019815    2561 scope.go:117] "RemoveContainer" containerID="02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31"
Jul 28 12:49:42.020095 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:49:42.020076    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31\": container with ID starting with 02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31 not found: ID does not exist" containerID="02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31"
Jul 28 12:49:42.020143 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.020101    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31"} err="failed to get container status \"02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31\": rpc error: code = NotFound desc = could not find container \"02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31\": container with ID starting with 02faa41bdf6f1af2a2f92788b256127051a3fb9bb1247ae7c9596ca290b59f31 not found: ID does not exist"
Jul 28 12:49:42.020143 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.020116    2561 scope.go:117] "RemoveContainer" containerID="dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8"
Jul 28 12:49:42.020361 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:49:42.020343    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8\": container with ID starting with dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8 not found: ID does not exist" containerID="dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8"
Jul 28 12:49:42.020424 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.020366    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8"} err="failed to get container status \"dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8\": rpc error: code = NotFound desc = could not find container \"dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8\": container with ID starting with dd023141475af5b33f72269085c14dfd8313a1d0b48ed99097d6aa7e4ebe39d8 not found: ID does not exist"
Jul 28 12:49:42.020704 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.020686    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"]
Jul 28 12:49:42.026456 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.026434    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-e5cf3-predictor-5755d65456-4ccgx"]
Jul 28 12:49:42.030294 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.030277    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10baf772-a265-4dbb-8afa-29fe5877f2e4-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:49:42.030356 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.030298    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:49:42.030356 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.030309    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10baf772-a265-4dbb-8afa-29fe5877f2e4-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:49:42.030356 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.030318    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10baf772-a265-4dbb-8afa-29fe5877f2e4-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:49:42.030356 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:42.030327    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kzcr\" (UniqueName: \"kubernetes.io/projected/10baf772-a265-4dbb-8afa-29fe5877f2e4-kube-api-access-9kzcr\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:49:43.003316 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:43.003227    2561 generic.go:356] "Generic (PLEG): container finished" podID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerID="14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95" exitCode=0
Jul 28 12:49:43.003316 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:43.003303    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" event={"ID":"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2","Type":"ContainerDied","Data":"14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95"}
Jul 28 12:49:43.101587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:43.101560    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" path="/var/lib/kubelet/pods/10baf772-a265-4dbb-8afa-29fe5877f2e4/volumes"
Jul 28 12:49:44.009266 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:44.009231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" event={"ID":"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2","Type":"ContainerStarted","Data":"be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd"}
Jul 28 12:49:44.009615 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:44.009297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" event={"ID":"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2","Type":"ContainerStarted","Data":"13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53"}
Jul 28 12:49:44.009615 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:44.009526    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:44.030629 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:44.030583    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podStartSLOduration=3.24876774 podStartE2EDuration="7.030568371s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.781800631s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:49:44.029339846 +0000 UTC m=+971.388944452" watchObservedRunningTime="2026-07-28 12:49:44.030568371 +0000 UTC m=+971.390172976"
Jul 28 12:49:45.013711 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:45.013680    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:45.014870 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:45.014846    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 12:49:46.017723 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:46.017682    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 12:49:51.022455 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:51.022425    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:49:51.022980 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:49:51.022950    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 12:50:01.023825 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:50:01.023715    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 12:50:11.023522 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:50:11.023482    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 12:50:21.023484 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:50:21.023438    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 12:50:31.023362 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:50:31.023322    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 12:50:41.023945 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:50:41.023898    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 12:50:51.022921 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:50:51.022879    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 12:51:01.023900 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:01.023871    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:51:17.372271 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:17.372235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:51:18.601078 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:18.601039    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:51:27.381536 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:27.381442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:51:27.530593 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:27.530547    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:51:27.535934 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:27.535910    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"]
Jul 28 12:51:27.536140 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:27.536116    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" containerID="cri-o://13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53" gracePeriod=30
Jul 28 12:51:27.536233 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:27.536198    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kube-rbac-proxy" containerID="cri-o://be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd" gracePeriod=30
Jul 28 12:51:27.631326 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:27.631278    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:51:27.695365 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:27.695328    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:51:27.953186 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:27.953083    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:51:28.403938 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:28.403905    2561 generic.go:356] "Generic (PLEG): container finished" podID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerID="be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd" exitCode=2
Jul 28 12:51:28.404315 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:28.403981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" event={"ID":"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2","Type":"ContainerDied","Data":"be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd"}
Jul 28 12:51:31.019025 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:31.018980    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 28 12:51:31.023270 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:31.023243    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 12:51:32.373606 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.373577    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:51:32.380464 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.380434    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-cabundle-cert\") pod \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") "
Jul 28 12:51:32.380593 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.380470    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-542g2\" (UniqueName: \"kubernetes.io/projected/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-api-access-542g2\") pod \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") "
Jul 28 12:51:32.380593 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.380494    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-rbac-proxy-sar-config\") pod \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") "
Jul 28 12:51:32.380593 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.380527    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-proxy-tls\") pod \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") "
Jul 28 12:51:32.380593 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.380563    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kserve-provision-location\") pod \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\" (UID: \"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2\") "
Jul 28 12:51:32.380944 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.380919    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" (UID: "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:51:32.381057 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.380940    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" (UID: "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:51:32.381057 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.380940    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" (UID: "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:51:32.382785 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.382755    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-api-access-542g2" (OuterVolumeSpecName: "kube-api-access-542g2") pod "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" (UID: "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2"). InnerVolumeSpecName "kube-api-access-542g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:51:32.382885 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.382809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" (UID: "fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:51:32.421215 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.421131    2561 generic.go:356] "Generic (PLEG): container finished" podID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerID="13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53" exitCode=0
Jul 28 12:51:32.421215 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.421177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" event={"ID":"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2","Type":"ContainerDied","Data":"13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53"}
Jul 28 12:51:32.421215 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.421196    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"
Jul 28 12:51:32.421429 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.421225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72" event={"ID":"fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2","Type":"ContainerDied","Data":"cc14ff15607a6d6f0485eded24ce3d1ff451dda79b959d1bce4ce535a34646ba"}
Jul 28 12:51:32.421429 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.421254    2561 scope.go:117] "RemoveContainer" containerID="be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd"
Jul 28 12:51:32.430143 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.430121    2561 scope.go:117] "RemoveContainer" containerID="13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53"
Jul 28 12:51:32.437911 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.437894    2561 scope.go:117] "RemoveContainer" containerID="14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95"
Jul 28 12:51:32.443902 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.443880    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"]
Jul 28 12:51:32.445844 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.445825    2561 scope.go:117] "RemoveContainer" containerID="be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd"
Jul 28 12:51:32.446214 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:51:32.446180    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd\": container with ID starting with be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd not found: ID does not exist" containerID="be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd"
Jul 28 12:51:32.446315 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.446228    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd"} err="failed to get container status \"be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd\": rpc error: code = NotFound desc = could not find container \"be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd\": container with ID starting with be24f8f9303546f3a3c9c7113a25f625004f36df1201b71d92d98aca4e3892cd not found: ID does not exist"
Jul 28 12:51:32.446315 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.446253    2561 scope.go:117] "RemoveContainer" containerID="13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53"
Jul 28 12:51:32.446560 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:51:32.446533    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53\": container with ID starting with 13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53 not found: ID does not exist" containerID="13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53"
Jul 28 12:51:32.446660 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.446566    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53"} err="failed to get container status \"13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53\": rpc error: code = NotFound desc = could not find container \"13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53\": container with ID starting with 13cd1d6a64feb90bb16a18b104367491c78b04e677355c087f373afc0f59ce53 not found: ID does not exist"
Jul 28 12:51:32.446660 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.446582    2561 scope.go:117] "RemoveContainer" containerID="14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95"
Jul 28 12:51:32.446877 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:51:32.446851    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95\": container with ID starting with 14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95 not found: ID does not exist" containerID="14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95"
Jul 28 12:51:32.446960 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.446884    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95"} err="failed to get container status \"14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95\": rpc error: code = NotFound desc = could not find container \"14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95\": container with ID starting with 14ef650ccdf56e2486e77e780b5248c38de24e812e297fbcc5fbdb102ddffc95 not found: ID does not exist"
Jul 28 12:51:32.447942 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.447922    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f8c4-predictor-645f5d9cb-7cn72"]
Jul 28 12:51:32.481231 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.481197    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:51:32.481231 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.481232    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:51:32.481397 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.481246    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-542g2\" (UniqueName: \"kubernetes.io/projected/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-api-access-542g2\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:51:32.481397 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.481260    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:51:32.481397 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:32.481272    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:51:33.105959 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:33.103451    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" path="/var/lib/kubelet/pods/fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2/volumes"
Jul 28 12:51:47.588080 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.588011    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:51:47.625793 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.625761    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"]
Jul 28 12:51:47.626053 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626040    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container"
Jul 28 12:51:47.626098 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626055    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container"
Jul 28 12:51:47.626098 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626071    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kube-rbac-proxy"
Jul 28 12:51:47.626098 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626077    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kube-rbac-proxy"
Jul 28 12:51:47.626098 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626083    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="storage-initializer"
Jul 28 12:51:47.626098 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626088    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="storage-initializer"
Jul 28 12:51:47.626098 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626096    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kube-rbac-proxy"
Jul 28 12:51:47.626098 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626101    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kube-rbac-proxy"
Jul 28 12:51:47.626300 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626110    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container"
Jul 28 12:51:47.626300 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626116    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container"
Jul 28 12:51:47.626300 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626122    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="storage-initializer"
Jul 28 12:51:47.626300 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626127    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="storage-initializer"
Jul 28 12:51:47.626300 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626193    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kube-rbac-proxy"
Jul 28 12:51:47.626300 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626200    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5354fb-a5a6-4e4f-b531-c31d6aadd0e2" containerName="kserve-container"
Jul 28 12:51:47.626300 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626206    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kube-rbac-proxy"
Jul 28 12:51:47.626300 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.626212    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="10baf772-a265-4dbb-8afa-29fe5877f2e4" containerName="kserve-container"
Jul 28 12:51:47.631148 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.631132    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.633623 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.633596    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-75f51-kube-rbac-proxy-sar-config\""
Jul 28 12:51:47.633819 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.633724    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-75f51-predictor-serving-cert\""
Jul 28 12:51:47.634129 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.634020    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 12:51:47.634129 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.634037    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 12:51:47.634129 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.634106    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rtw6d\""
Jul 28 12:51:47.634326 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.634036    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 12:51:47.634326 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.634106    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 12:51:47.643421 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.643392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"]
Jul 28 12:51:47.680251 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.680217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.680251 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.680257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/670f0f96-c8d1-4222-b207-f0d486477fa4-proxy-tls\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.680513 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.680275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppp6\" (UniqueName: \"kubernetes.io/projected/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-api-access-kppp6\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.680513 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.680328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/670f0f96-c8d1-4222-b207-f0d486477fa4-kserve-provision-location\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.680513 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.680355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-cabundle-cert\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.744148 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.744109    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:51:47.781230 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.781187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.781230 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.781232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/670f0f96-c8d1-4222-b207-f0d486477fa4-proxy-tls\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.781485 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.781253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppp6\" (UniqueName: \"kubernetes.io/projected/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-api-access-kppp6\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.781485 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.781365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/670f0f96-c8d1-4222-b207-f0d486477fa4-kserve-provision-location\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.781485 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.781428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-cabundle-cert\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.781775 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.781756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/670f0f96-c8d1-4222-b207-f0d486477fa4-kserve-provision-location\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.782011 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.781991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.782108 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.782090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-cabundle-cert\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.783894 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.783877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/670f0f96-c8d1-4222-b207-f0d486477fa4-proxy-tls\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.790180 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.790159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppp6\" (UniqueName: \"kubernetes.io/projected/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-api-access-kppp6\") pod \"isvc-logger-raw-75f51-predictor-6f9549fc45-7k585\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") " pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:47.947005 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:47.946916    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:48.082899 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:48.082874    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"]
Jul 28 12:51:48.087695 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:51:48.087664    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670f0f96_c8d1_4222_b207_f0d486477fa4.slice/crio-9ecb071acdb66e0149c0c345d79065dc6d54fe5319c8f6f44f3b9c9a449c4dd7 WatchSource:0}: Error finding container 9ecb071acdb66e0149c0c345d79065dc6d54fe5319c8f6f44f3b9c9a449c4dd7: Status 404 returned error can't find the container with id 9ecb071acdb66e0149c0c345d79065dc6d54fe5319c8f6f44f3b9c9a449c4dd7
Jul 28 12:51:48.483789 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:48.483747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" event={"ID":"670f0f96-c8d1-4222-b207-f0d486477fa4","Type":"ContainerStarted","Data":"e9afec77c7a5102607e269e9fe08c0f435fbaa49094e92cefdc5764933a3e217"}
Jul 28 12:51:48.483967 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:48.483831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" event={"ID":"670f0f96-c8d1-4222-b207-f0d486477fa4","Type":"ContainerStarted","Data":"9ecb071acdb66e0149c0c345d79065dc6d54fe5319c8f6f44f3b9c9a449c4dd7"}
Jul 28 12:51:51.611747 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:51.611694    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:51:52.500870 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:52.500833    2561 generic.go:356] "Generic (PLEG): container finished" podID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerID="e9afec77c7a5102607e269e9fe08c0f435fbaa49094e92cefdc5764933a3e217" exitCode=0
Jul 28 12:51:52.501015 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:52.500905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" event={"ID":"670f0f96-c8d1-4222-b207-f0d486477fa4","Type":"ContainerDied","Data":"e9afec77c7a5102607e269e9fe08c0f435fbaa49094e92cefdc5764933a3e217"}
Jul 28 12:51:53.508038 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:53.508004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" event={"ID":"670f0f96-c8d1-4222-b207-f0d486477fa4","Type":"ContainerStarted","Data":"efc29cce1960b73c5beb5f1c82d6a12417f7f5beba9ebdfecca8b3b698d646bf"}
Jul 28 12:51:53.508478 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:53.508084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" event={"ID":"670f0f96-c8d1-4222-b207-f0d486477fa4","Type":"ContainerStarted","Data":"494fe2e5e718b204e22d41987322e3cd5522190e2e40958b4a562544c8f88b34"}
Jul 28 12:51:53.508478 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:53.508125    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" event={"ID":"670f0f96-c8d1-4222-b207-f0d486477fa4","Type":"ContainerStarted","Data":"54e6ab07db37d4d419cb20d12ec0a468560bd1d03da86584dd7999945ce7fe4d"}
Jul 28 12:51:53.508478 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:53.508421    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:53.508478 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:53.508467    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:53.508631 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:53.508497    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:53.510109 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:53.510077    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:51:53.510751 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:53.510710    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:51:53.528856 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:53.528816    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podStartSLOduration=2.854436761 podStartE2EDuration="6.528802841s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.67436608s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:51:53.526879404 +0000 UTC m=+1100.886484009" watchObservedRunningTime="2026-07-28 12:51:53.528802841 +0000 UTC m=+1100.888407446"
Jul 28 12:51:54.512023 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:54.511977    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:51:54.512473 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:54.512445    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:51:59.516187 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:59.516159    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:51:59.516783 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:59.516754    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:51:59.517184 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:51:59.517161    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:52:09.517161 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:09.517117    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:52:09.517610 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:09.517583    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:52:19.517349 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:19.517307    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:52:19.517823 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:19.517771    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:52:29.516972 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:29.516927    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:52:29.517405 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:29.517381    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:52:39.516977 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:39.516938    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:52:39.517554 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:39.517503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:52:49.517028 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:49.516984    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:52:49.517587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:49.517565    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:52:59.517189 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:59.517100    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:52:59.517594 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:52:59.517526    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:53:09.102036 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:09.102009    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:53:09.102414 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:09.102096    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:53:22.879355 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:22.879321    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"]
Jul 28 12:53:22.879905 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:22.879597    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" containerID="cri-o://54e6ab07db37d4d419cb20d12ec0a468560bd1d03da86584dd7999945ce7fe4d" gracePeriod=30
Jul 28 12:53:22.879905 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:22.879632    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kube-rbac-proxy" containerID="cri-o://494fe2e5e718b204e22d41987322e3cd5522190e2e40958b4a562544c8f88b34" gracePeriod=30
Jul 28 12:53:22.879905 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:22.879631    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" containerID="cri-o://efc29cce1960b73c5beb5f1c82d6a12417f7f5beba9ebdfecca8b3b698d646bf" gracePeriod=30
Jul 28 12:53:22.927270 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:22.927226    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:53:22.954766 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:22.954715    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"]
Jul 28 12:53:22.958469 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:22.958449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:22.960864 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:22.960847    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-b38ea-predictor-serving-cert\""
Jul 28 12:53:22.960972 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:22.960848    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-b38ea-kube-rbac-proxy-sar-config\""
Jul 28 12:53:22.970664 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:22.970644    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"]
Jul 28 12:53:23.006704 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.006673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:53:23.028062 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.028033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a41a6f4b-61d5-47da-9816-15287c4daf00-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.028278 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.028077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb777\" (UniqueName: \"kubernetes.io/projected/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-api-access-sb777\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.028278 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.028102    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a41a6f4b-61d5-47da-9816-15287c4daf00-proxy-tls\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.028278 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.028200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.028278 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.028232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.129017 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.128990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.129206 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.129023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.129206 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.129051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a41a6f4b-61d5-47da-9816-15287c4daf00-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.129303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.129219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb777\" (UniqueName: \"kubernetes.io/projected/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-api-access-sb777\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.129303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.129270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a41a6f4b-61d5-47da-9816-15287c4daf00-proxy-tls\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.129433 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.129355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a41a6f4b-61d5-47da-9816-15287c4daf00-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.129695 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.129673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.129809 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.129756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.131942 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.131922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a41a6f4b-61d5-47da-9816-15287c4daf00-proxy-tls\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.137624 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.137604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb777\" (UniqueName: \"kubernetes.io/projected/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-api-access-sb777\") pod \"isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.268534 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.268499    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:23.403593 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.403571    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"]
Jul 28 12:53:23.406035 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:53:23.406015    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41a6f4b_61d5_47da_9816_15287c4daf00.slice/crio-95b842c65eac434193adbc4f492c27e3b0f8f6241d5f9677aec5cf5ec9c29fcc WatchSource:0}: Error finding container 95b842c65eac434193adbc4f492c27e3b0f8f6241d5f9677aec5cf5ec9c29fcc: Status 404 returned error can't find the container with id 95b842c65eac434193adbc4f492c27e3b0f8f6241d5f9677aec5cf5ec9c29fcc
Jul 28 12:53:23.426119 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.426098    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:53:23.853921 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.853879    2561 generic.go:356] "Generic (PLEG): container finished" podID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerID="494fe2e5e718b204e22d41987322e3cd5522190e2e40958b4a562544c8f88b34" exitCode=2
Jul 28 12:53:23.854116 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.853944    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" event={"ID":"670f0f96-c8d1-4222-b207-f0d486477fa4","Type":"ContainerDied","Data":"494fe2e5e718b204e22d41987322e3cd5522190e2e40958b4a562544c8f88b34"}
Jul 28 12:53:23.855495 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.855472    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" event={"ID":"a41a6f4b-61d5-47da-9816-15287c4daf00","Type":"ContainerStarted","Data":"9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0"}
Jul 28 12:53:23.855619 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:23.855519    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" event={"ID":"a41a6f4b-61d5-47da-9816-15287c4daf00","Type":"ContainerStarted","Data":"95b842c65eac434193adbc4f492c27e3b0f8f6241d5f9677aec5cf5ec9c29fcc"}
Jul 28 12:53:24.512679 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:24.512632    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 28 12:53:27.879576 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:27.879542    2561 generic.go:356] "Generic (PLEG): container finished" podID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerID="54e6ab07db37d4d419cb20d12ec0a468560bd1d03da86584dd7999945ce7fe4d" exitCode=0
Jul 28 12:53:27.880065 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:27.879608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" event={"ID":"670f0f96-c8d1-4222-b207-f0d486477fa4","Type":"ContainerDied","Data":"54e6ab07db37d4d419cb20d12ec0a468560bd1d03da86584dd7999945ce7fe4d"}
Jul 28 12:53:27.881287 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:27.881265    2561 generic.go:356] "Generic (PLEG): container finished" podID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerID="9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0" exitCode=0
Jul 28 12:53:27.881422 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:27.881337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" event={"ID":"a41a6f4b-61d5-47da-9816-15287c4daf00","Type":"ContainerDied","Data":"9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0"}
Jul 28 12:53:28.887573 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:28.887537    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" event={"ID":"a41a6f4b-61d5-47da-9816-15287c4daf00","Type":"ContainerStarted","Data":"d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4"}
Jul 28 12:53:28.887975 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:28.887602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" event={"ID":"a41a6f4b-61d5-47da-9816-15287c4daf00","Type":"ContainerStarted","Data":"17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb"}
Jul 28 12:53:28.887975 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:28.887886    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:28.887975 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:28.887940    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:28.889274 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:28.889248    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:53:28.908992 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:28.908945    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podStartSLOduration=3.294131208 podStartE2EDuration="6.908931918s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.61480071s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:28.90685812 +0000 UTC m=+1196.266462726" watchObservedRunningTime="2026-07-28 12:53:28.908931918 +0000 UTC m=+1196.268536519"
Jul 28 12:53:29.098068 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:29.098024    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:53:29.098353 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:29.098319    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:53:29.513080 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:29.513033    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 28 12:53:29.891588 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:29.891547    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:53:33.064928 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:33.064903    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:53:33.066099 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:33.066072    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:53:34.512989 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:34.512953    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 28 12:53:34.513439 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:34.513101    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:53:34.895918 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:34.895889    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:53:34.896399 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:34.896372    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:53:39.097796 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:39.097727    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:53:39.098186 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:39.097998    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:53:39.512767 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:39.512650    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 28 12:53:44.512190 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:44.512138    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 28 12:53:44.896856 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:44.896818    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:53:49.098552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:49.098503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 12:53:49.099070 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:49.098865    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:53:49.102111 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:49.102085    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:53:49.102229 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:49.102176    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:53:49.512574 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:49.512479    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 28 12:53:52.982669 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:52.982638    2561 generic.go:356] "Generic (PLEG): container finished" podID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerID="efc29cce1960b73c5beb5f1c82d6a12417f7f5beba9ebdfecca8b3b698d646bf" exitCode=0
Jul 28 12:53:52.983001 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:52.982749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" event={"ID":"670f0f96-c8d1-4222-b207-f0d486477fa4","Type":"ContainerDied","Data":"efc29cce1960b73c5beb5f1c82d6a12417f7f5beba9ebdfecca8b3b698d646bf"}
Jul 28 12:53:53.024347 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.024316    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:53:53.177228 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.177152    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-cabundle-cert\") pod \"670f0f96-c8d1-4222-b207-f0d486477fa4\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") "
Jul 28 12:53:53.177372 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.177225    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-rbac-proxy-sar-config\") pod \"670f0f96-c8d1-4222-b207-f0d486477fa4\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") "
Jul 28 12:53:53.177372 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.177255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/670f0f96-c8d1-4222-b207-f0d486477fa4-proxy-tls\") pod \"670f0f96-c8d1-4222-b207-f0d486477fa4\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") "
Jul 28 12:53:53.177372 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.177299    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kppp6\" (UniqueName: \"kubernetes.io/projected/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-api-access-kppp6\") pod \"670f0f96-c8d1-4222-b207-f0d486477fa4\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") "
Jul 28 12:53:53.177372 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.177338    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/670f0f96-c8d1-4222-b207-f0d486477fa4-kserve-provision-location\") pod \"670f0f96-c8d1-4222-b207-f0d486477fa4\" (UID: \"670f0f96-c8d1-4222-b207-f0d486477fa4\") "
Jul 28 12:53:53.177606 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.177573    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "670f0f96-c8d1-4222-b207-f0d486477fa4" (UID: "670f0f96-c8d1-4222-b207-f0d486477fa4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:53:53.177797 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.177586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "670f0f96-c8d1-4222-b207-f0d486477fa4" (UID: "670f0f96-c8d1-4222-b207-f0d486477fa4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:53:53.177866 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.177804    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670f0f96-c8d1-4222-b207-f0d486477fa4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "670f0f96-c8d1-4222-b207-f0d486477fa4" (UID: "670f0f96-c8d1-4222-b207-f0d486477fa4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:53:53.179537 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.179512    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670f0f96-c8d1-4222-b207-f0d486477fa4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "670f0f96-c8d1-4222-b207-f0d486477fa4" (UID: "670f0f96-c8d1-4222-b207-f0d486477fa4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:53:53.179591 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.179564    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-api-access-kppp6" (OuterVolumeSpecName: "kube-api-access-kppp6") pod "670f0f96-c8d1-4222-b207-f0d486477fa4" (UID: "670f0f96-c8d1-4222-b207-f0d486477fa4"). InnerVolumeSpecName "kube-api-access-kppp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:53:53.278350 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.278315    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:53:53.278350 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.278344    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/670f0f96-c8d1-4222-b207-f0d486477fa4-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:53:53.278350 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.278355    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kppp6\" (UniqueName: \"kubernetes.io/projected/670f0f96-c8d1-4222-b207-f0d486477fa4-kube-api-access-kppp6\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:53:53.278564 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.278363    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/670f0f96-c8d1-4222-b207-f0d486477fa4-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:53:53.278564 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.278372    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/670f0f96-c8d1-4222-b207-f0d486477fa4-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:53:53.988944 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.988905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585" event={"ID":"670f0f96-c8d1-4222-b207-f0d486477fa4","Type":"ContainerDied","Data":"9ecb071acdb66e0149c0c345d79065dc6d54fe5319c8f6f44f3b9c9a449c4dd7"}
Jul 28 12:53:53.989427 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.988999    2561 scope.go:117] "RemoveContainer" containerID="efc29cce1960b73c5beb5f1c82d6a12417f7f5beba9ebdfecca8b3b698d646bf"
Jul 28 12:53:53.989427 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.989007    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"
Jul 28 12:53:53.998489 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:53.998472    2561 scope.go:117] "RemoveContainer" containerID="494fe2e5e718b204e22d41987322e3cd5522190e2e40958b4a562544c8f88b34"
Jul 28 12:53:54.005806 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:54.005785    2561 scope.go:117] "RemoveContainer" containerID="54e6ab07db37d4d419cb20d12ec0a468560bd1d03da86584dd7999945ce7fe4d"
Jul 28 12:53:54.011233 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:54.011214    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"]
Jul 28 12:53:54.013596 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:54.013573    2561 scope.go:117] "RemoveContainer" containerID="e9afec77c7a5102607e269e9fe08c0f435fbaa49094e92cefdc5764933a3e217"
Jul 28 12:53:54.015768 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:54.015726    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-75f51-predictor-6f9549fc45-7k585"]
Jul 28 12:53:54.896783 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:54.896749    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:53:55.102287 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:53:55.102256    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" path="/var/lib/kubelet/pods/670f0f96-c8d1-4222-b207-f0d486477fa4/volumes"
Jul 28 12:54:04.238578 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:54:04.238544    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:04.897104 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:54:04.897063    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:54:14.896421 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:54:14.896382    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:54:24.896884 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:54:24.896797    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:54:34.897349 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:54:34.897307    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:54:43.098712 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:54:43.098664    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:54:53.098867 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:54:53.098827    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:55:03.099041 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:55:03.099001    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:55:13.105944 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:55:13.105905    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:55:18.106646 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:55:18.106593    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:23.099328 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:55:23.099270    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:55:33.099008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:55:33.098969    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:55:43.099352 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:55:43.099307    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:55:53.099366 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:55:53.099320    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:56:03.101895 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:03.101863    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:56:13.175176 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:13.175140    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"]
Jul 28 12:56:13.175610 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:13.175400    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" containerID="cri-o://17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb" gracePeriod=30
Jul 28 12:56:13.175610 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:13.175507    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kube-rbac-proxy" containerID="cri-o://d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4" gracePeriod=30
Jul 28 12:56:13.207156 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:13.207106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:13.531526 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:13.531498    2561 generic.go:356] "Generic (PLEG): container finished" podID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerID="d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4" exitCode=2
Jul 28 12:56:13.531704 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:13.531538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" event={"ID":"a41a6f4b-61d5-47da-9816-15287c4daf00","Type":"ContainerDied","Data":"d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4"}
Jul 28 12:56:14.892393 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:14.892349    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 28 12:56:18.283337 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.283249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:18.339115 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339079    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"]
Jul 28 12:56:18.339496 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339478    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="storage-initializer"
Jul 28 12:56:18.339587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339499    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="storage-initializer"
Jul 28 12:56:18.339587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339515    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container"
Jul 28 12:56:18.339587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339524    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container"
Jul 28 12:56:18.339587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339533    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kube-rbac-proxy"
Jul 28 12:56:18.339587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339543    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kube-rbac-proxy"
Jul 28 12:56:18.339587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339574    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent"
Jul 28 12:56:18.339587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339582    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent"
Jul 28 12:56:18.340162 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339657    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kube-rbac-proxy"
Jul 28 12:56:18.340162 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339671    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="agent"
Jul 28 12:56:18.340162 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.339681    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="670f0f96-c8d1-4222-b207-f0d486477fa4" containerName="kserve-container"
Jul 28 12:56:18.342973 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.342954    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.345616 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.345593    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 12:56:18.345769 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.345593    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 12:56:18.355205 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.355183    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"]
Jul 28 12:56:18.356996 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.356973    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"]
Jul 28 12:56:18.360192 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.360176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.362699 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.362686    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 12:56:18.374683 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.374657    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"]
Jul 28 12:56:18.377762 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.377715    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:18.439103 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.439072    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:18.448801 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.448771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmjq\" (UniqueName: \"kubernetes.io/projected/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-api-access-plmjq\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.448907 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.448820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f5131c5-4d1a-40a1-b48c-8889059668b2-proxy-tls\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.448907 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.448850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq8m8\" (UniqueName: \"kubernetes.io/projected/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-api-access-tq8m8\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.449000 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.448921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-cabundle-cert\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.449000 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.448957    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.449000 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.448993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.449125 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.449017    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d75f4d5d-66c3-435b-9dbd-5cd464123c42-proxy-tls\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.449125 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.449082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.449125 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.449106    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f5131c5-4d1a-40a1-b48c-8889059668b2-kserve-provision-location\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.449249 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.449132    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.550146 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.550146 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f5131c5-4d1a-40a1-b48c-8889059668b2-kserve-provision-location\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.550383 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.550383 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmjq\" (UniqueName: \"kubernetes.io/projected/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-api-access-plmjq\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.550383 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f5131c5-4d1a-40a1-b48c-8889059668b2-proxy-tls\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.550549 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq8m8\" (UniqueName: \"kubernetes.io/projected/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-api-access-tq8m8\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.550549 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-cabundle-cert\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.550549 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.550549 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.550549 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f5131c5-4d1a-40a1-b48c-8889059668b2-kserve-provision-location\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.550549 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d75f4d5d-66c3-435b-9dbd-5cd464123c42-proxy-tls\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.550890 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:56:18.550607    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 12:56:18.550890 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:56:18.550666    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d75f4d5d-66c3-435b-9dbd-5cd464123c42-proxy-tls podName:d75f4d5d-66c3-435b-9dbd-5cd464123c42 nodeName:}" failed. No retries permitted until 2026-07-28 12:56:19.050646199 +0000 UTC m=+1366.410250785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d75f4d5d-66c3-435b-9dbd-5cd464123c42-proxy-tls") pod "isvc-canary-create-v2-predictor-69987fd58d-lcbsd" (UID: "d75f4d5d-66c3-435b-9dbd-5cd464123c42") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 12:56:18.551008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.550893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.551309 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.551225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-cabundle-cert\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.551309 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.551258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.551578 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.551546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.551775 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.551757    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.553846 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.553821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f5131c5-4d1a-40a1-b48c-8889059668b2-proxy-tls\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.559519 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.559497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmjq\" (UniqueName: \"kubernetes.io/projected/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-api-access-plmjq\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:18.562326 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.562301    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq8m8\" (UniqueName: \"kubernetes.io/projected/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-api-access-tq8m8\") pod \"isvc-canary-create-predictor-5658fb66fc-bss54\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.653576 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.653534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:18.672220 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.672182    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:18.795004 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:18.794979    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"]
Jul 28 12:56:18.797145 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:56:18.797125    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5131c5_4d1a_40a1_b48c_8889059668b2.slice/crio-b73812570311f6e72c44d82f468a71ccd9bdc83bdd1a1f793211027c89f1d3eb WatchSource:0}: Error finding container b73812570311f6e72c44d82f468a71ccd9bdc83bdd1a1f793211027c89f1d3eb: Status 404 returned error can't find the container with id b73812570311f6e72c44d82f468a71ccd9bdc83bdd1a1f793211027c89f1d3eb
Jul 28 12:56:19.054299 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:19.054194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d75f4d5d-66c3-435b-9dbd-5cd464123c42-proxy-tls\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:19.056677 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:19.056649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d75f4d5d-66c3-435b-9dbd-5cd464123c42-proxy-tls\") pod \"isvc-canary-create-v2-predictor-69987fd58d-lcbsd\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:19.271239 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:19.271208    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:19.413849 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:19.413814    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"]
Jul 28 12:56:19.417119 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:56:19.417092    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75f4d5d_66c3_435b_9dbd_5cd464123c42.slice/crio-1fb8fccda44271523ec1e7fbcfab1eeca221a4a724a4f323724192c0f947df06 WatchSource:0}: Error finding container 1fb8fccda44271523ec1e7fbcfab1eeca221a4a724a4f323724192c0f947df06: Status 404 returned error can't find the container with id 1fb8fccda44271523ec1e7fbcfab1eeca221a4a724a4f323724192c0f947df06
Jul 28 12:56:19.560775 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:19.560716    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" event={"ID":"9f5131c5-4d1a-40a1-b48c-8889059668b2","Type":"ContainerStarted","Data":"58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7"}
Jul 28 12:56:19.560960 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:19.560807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" event={"ID":"9f5131c5-4d1a-40a1-b48c-8889059668b2","Type":"ContainerStarted","Data":"b73812570311f6e72c44d82f468a71ccd9bdc83bdd1a1f793211027c89f1d3eb"}
Jul 28 12:56:19.562251 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:19.562224    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" event={"ID":"d75f4d5d-66c3-435b-9dbd-5cd464123c42","Type":"ContainerStarted","Data":"332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c"}
Jul 28 12:56:19.562356 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:19.562279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" event={"ID":"d75f4d5d-66c3-435b-9dbd-5cd464123c42","Type":"ContainerStarted","Data":"1fb8fccda44271523ec1e7fbcfab1eeca221a4a724a4f323724192c0f947df06"}
Jul 28 12:56:19.892544 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:19.892494    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 28 12:56:23.098697 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.098654    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 12:56:23.527064 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.527037    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:56:23.580420 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.580389    2561 generic.go:356] "Generic (PLEG): container finished" podID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerID="17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb" exitCode=0
Jul 28 12:56:23.580613 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.580436    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" event={"ID":"a41a6f4b-61d5-47da-9816-15287c4daf00","Type":"ContainerDied","Data":"17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb"}
Jul 28 12:56:23.580613 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.580451    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"
Jul 28 12:56:23.580613 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.580503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp" event={"ID":"a41a6f4b-61d5-47da-9816-15287c4daf00","Type":"ContainerDied","Data":"95b842c65eac434193adbc4f492c27e3b0f8f6241d5f9677aec5cf5ec9c29fcc"}
Jul 28 12:56:23.580613 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.580542    2561 scope.go:117] "RemoveContainer" containerID="d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4"
Jul 28 12:56:23.582067 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.582040    2561 generic.go:356] "Generic (PLEG): container finished" podID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerID="332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c" exitCode=0
Jul 28 12:56:23.582194 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.582121    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" event={"ID":"d75f4d5d-66c3-435b-9dbd-5cd464123c42","Type":"ContainerDied","Data":"332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c"}
Jul 28 12:56:23.583598 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.583580    2561 generic.go:356] "Generic (PLEG): container finished" podID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerID="58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7" exitCode=0
Jul 28 12:56:23.583674 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.583628    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" event={"ID":"9f5131c5-4d1a-40a1-b48c-8889059668b2","Type":"ContainerDied","Data":"58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7"}
Jul 28 12:56:23.591253 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.591231    2561 scope.go:117] "RemoveContainer" containerID="17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb"
Jul 28 12:56:23.594677 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.594655    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a41a6f4b-61d5-47da-9816-15287c4daf00-proxy-tls\") pod \"a41a6f4b-61d5-47da-9816-15287c4daf00\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") "
Jul 28 12:56:23.594787 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.594725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a41a6f4b-61d5-47da-9816-15287c4daf00-kserve-provision-location\") pod \"a41a6f4b-61d5-47da-9816-15287c4daf00\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") "
Jul 28 12:56:23.594849 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.594835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-cabundle-cert\") pod \"a41a6f4b-61d5-47da-9816-15287c4daf00\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") "
Jul 28 12:56:23.594894 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.594870    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-rbac-proxy-sar-config\") pod \"a41a6f4b-61d5-47da-9816-15287c4daf00\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") "
Jul 28 12:56:23.594943 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.594907    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb777\" (UniqueName: \"kubernetes.io/projected/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-api-access-sb777\") pod \"a41a6f4b-61d5-47da-9816-15287c4daf00\" (UID: \"a41a6f4b-61d5-47da-9816-15287c4daf00\") "
Jul 28 12:56:23.595117 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.595020    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41a6f4b-61d5-47da-9816-15287c4daf00-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a41a6f4b-61d5-47da-9816-15287c4daf00" (UID: "a41a6f4b-61d5-47da-9816-15287c4daf00"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:56:23.595215 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.595171    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a41a6f4b-61d5-47da-9816-15287c4daf00" (UID: "a41a6f4b-61d5-47da-9816-15287c4daf00"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:23.595299 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.595215    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a41a6f4b-61d5-47da-9816-15287c4daf00" (UID: "a41a6f4b-61d5-47da-9816-15287c4daf00"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:23.597064 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.597036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41a6f4b-61d5-47da-9816-15287c4daf00-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a41a6f4b-61d5-47da-9816-15287c4daf00" (UID: "a41a6f4b-61d5-47da-9816-15287c4daf00"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:23.597415 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.597398    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-api-access-sb777" (OuterVolumeSpecName: "kube-api-access-sb777") pod "a41a6f4b-61d5-47da-9816-15287c4daf00" (UID: "a41a6f4b-61d5-47da-9816-15287c4daf00"). InnerVolumeSpecName "kube-api-access-sb777". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:56:23.601966 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.601946    2561 scope.go:117] "RemoveContainer" containerID="9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0"
Jul 28 12:56:23.611907 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.611887    2561 scope.go:117] "RemoveContainer" containerID="d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4"
Jul 28 12:56:23.612182 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:56:23.612162    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4\": container with ID starting with d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4 not found: ID does not exist" containerID="d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4"
Jul 28 12:56:23.612263 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.612196    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4"} err="failed to get container status \"d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4\": rpc error: code = NotFound desc = could not find container \"d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4\": container with ID starting with d65bad4a19961cc50ade700c04b46a8ff8588ed27d413f40a109f2335fd141e4 not found: ID does not exist"
Jul 28 12:56:23.612263 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.612221    2561 scope.go:117] "RemoveContainer" containerID="17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb"
Jul 28 12:56:23.612522 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:56:23.612500    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb\": container with ID starting with 17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb not found: ID does not exist" containerID="17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb"
Jul 28 12:56:23.612563 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.612530    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb"} err="failed to get container status \"17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb\": rpc error: code = NotFound desc = could not find container \"17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb\": container with ID starting with 17f6fbb8167e8ef44227b5c042b363816a8e0da200f6b2c8c1bc3bba365830cb not found: ID does not exist"
Jul 28 12:56:23.612563 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.612547    2561 scope.go:117] "RemoveContainer" containerID="9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0"
Jul 28 12:56:23.612773 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:56:23.612751    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0\": container with ID starting with 9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0 not found: ID does not exist" containerID="9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0"
Jul 28 12:56:23.612866 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.612774    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0"} err="failed to get container status \"9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0\": rpc error: code = NotFound desc = could not find container \"9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0\": container with ID starting with 9256f5d387b9e33931393558588d6b0c9c589eb822f1e07bcf3056a7e4f40fb0 not found: ID does not exist"
Jul 28 12:56:23.695690 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.695641    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:56:23.695690 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.695681    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb777\" (UniqueName: \"kubernetes.io/projected/a41a6f4b-61d5-47da-9816-15287c4daf00-kube-api-access-sb777\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:56:23.695690 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.695699    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a41a6f4b-61d5-47da-9816-15287c4daf00-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:56:23.696001 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.695714    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a41a6f4b-61d5-47da-9816-15287c4daf00-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:56:23.696001 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.695759    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a41a6f4b-61d5-47da-9816-15287c4daf00-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:56:23.903590 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.903555    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"]
Jul 28 12:56:23.907644 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:23.907618    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b38ea-predictor-77f776f9b6-jfbkp"]
Jul 28 12:56:24.595314 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:24.595248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" event={"ID":"9f5131c5-4d1a-40a1-b48c-8889059668b2","Type":"ContainerStarted","Data":"e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700"}
Jul 28 12:56:24.595782 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:24.595425    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" event={"ID":"9f5131c5-4d1a-40a1-b48c-8889059668b2","Type":"ContainerStarted","Data":"c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4"}
Jul 28 12:56:24.595885 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:24.595866    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:24.596069 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:24.596048    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:24.597452 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:24.597424    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 12:56:24.598634 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:24.598606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" event={"ID":"d75f4d5d-66c3-435b-9dbd-5cd464123c42","Type":"ContainerStarted","Data":"8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6"}
Jul 28 12:56:24.598750 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:24.598654    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" event={"ID":"d75f4d5d-66c3-435b-9dbd-5cd464123c42","Type":"ContainerStarted","Data":"5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703"}
Jul 28 12:56:24.598855 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:24.598839    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:24.619323 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:24.619264    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podStartSLOduration=2.861231825 podStartE2EDuration="6.619249612s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.758017787s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:56:24.618313088 +0000 UTC m=+1371.977917693" watchObservedRunningTime="2026-07-28 12:56:24.619249612 +0000 UTC m=+1371.978854217"
Jul 28 12:56:24.637846 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:24.637791    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podStartSLOduration=2.733642825 podStartE2EDuration="6.637773617s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.904130792s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:56:24.636883472 +0000 UTC m=+1371.996488083" watchObservedRunningTime="2026-07-28 12:56:24.637773617 +0000 UTC m=+1371.997378218"
Jul 28 12:56:25.102940 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:25.102903    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" path="/var/lib/kubelet/pods/a41a6f4b-61d5-47da-9816-15287c4daf00/volumes"
Jul 28 12:56:25.603147 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:25.603094    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 12:56:25.603586 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:25.603119    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:25.604194 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:25.604170    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 12:56:26.606957 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:26.606916    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 12:56:30.607367 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:30.607335    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:56:30.607878 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:30.607851    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 12:56:31.611416 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:31.611386    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:56:31.612085 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:31.612055    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 12:56:40.608787 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:40.608717    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 12:56:41.612928 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:41.612891    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 12:56:50.607956 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:50.607917    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 12:56:51.612335 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:56:51.612294    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 12:57:00.608438 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:00.608394    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 12:57:01.612817 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:01.612781    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 12:57:10.608137 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:10.608098    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 12:57:11.612921 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:11.612877    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 12:57:20.607897 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:20.607856    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 12:57:21.612303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:21.612253    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 12:57:30.608603 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:30.608520    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 12:57:31.612951 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:31.612907    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 12:57:40.608656 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:40.608626    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:57:41.612934 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:41.612904    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:57:48.437748 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.437697    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"]
Jul 28 12:57:48.438221 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.438097    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" containerID="cri-o://c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4" gracePeriod=30
Jul 28 12:57:48.438221 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.438153    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kube-rbac-proxy" containerID="cri-o://e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700" gracePeriod=30
Jul 28 12:57:48.442916 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.442886    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"]
Jul 28 12:57:48.443277 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.443248    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kube-rbac-proxy" containerID="cri-o://8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6" gracePeriod=30
Jul 28 12:57:48.443375 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.443234    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" containerID="cri-o://5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703" gracePeriod=30
Jul 28 12:57:48.465214 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.465189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:48.566381 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.566351    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:48.939391 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.939356    2561 generic.go:356] "Generic (PLEG): container finished" podID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerID="8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6" exitCode=2
Jul 28 12:57:48.939595 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.939432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" event={"ID":"d75f4d5d-66c3-435b-9dbd-5cd464123c42","Type":"ContainerDied","Data":"8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6"}
Jul 28 12:57:48.941356 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.941333    2561 generic.go:356] "Generic (PLEG): container finished" podID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerID="e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700" exitCode=2
Jul 28 12:57:48.941487 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:48.941406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" event={"ID":"9f5131c5-4d1a-40a1-b48c-8889059668b2","Type":"ContainerDied","Data":"e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700"}
Jul 28 12:57:50.603917 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:50.603867    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 12:57:50.608214 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:50.608190    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 12:57:51.608136 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:51.608089    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 12:57:51.612485 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:51.612454    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 12:57:53.552357 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.552219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:53.612167 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.612091    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:57:53.615212 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615187    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"]
Jul 28 12:57:53.615504 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615490    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="storage-initializer"
Jul 28 12:57:53.615504 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615504    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="storage-initializer"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615525    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615531    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615538    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615544    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615551    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kube-rbac-proxy"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615555    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kube-rbac-proxy"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615565    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kube-rbac-proxy"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615570    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kube-rbac-proxy"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615578    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="storage-initializer"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="storage-initializer"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615636    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kserve-container"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615645    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kube-rbac-proxy"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615651    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a41a6f4b-61d5-47da-9816-15287c4daf00" containerName="kube-rbac-proxy"
Jul 28 12:57:53.615651 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.615657    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerName="kserve-container"
Jul 28 12:57:53.618564 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.618543    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.622831 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.621781    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 12:57:53.622831 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.622016    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 12:57:53.623013 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.622889    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:53.623368 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.623342    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"]
Jul 28 12:57:53.628159 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.627799    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.631510 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.630696    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"]
Jul 28 12:57:53.631510 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.630891    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 12:57:53.645903 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.645864    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"]
Jul 28 12:57:53.710523 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.710499    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:57:53.734775 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.734192    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:53.768073 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768035    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq8m8\" (UniqueName: \"kubernetes.io/projected/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-api-access-tq8m8\") pod \"9f5131c5-4d1a-40a1-b48c-8889059668b2\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") "
Jul 28 12:57:53.768284 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768113    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-rbac-proxy-sar-config\") pod \"9f5131c5-4d1a-40a1-b48c-8889059668b2\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") "
Jul 28 12:57:53.768284 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768170    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f5131c5-4d1a-40a1-b48c-8889059668b2-proxy-tls\") pod \"9f5131c5-4d1a-40a1-b48c-8889059668b2\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") "
Jul 28 12:57:53.768284 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768190    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-cabundle-cert\") pod \"9f5131c5-4d1a-40a1-b48c-8889059668b2\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") "
Jul 28 12:57:53.768284 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768206    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f5131c5-4d1a-40a1-b48c-8889059668b2-kserve-provision-location\") pod \"9f5131c5-4d1a-40a1-b48c-8889059668b2\" (UID: \"9f5131c5-4d1a-40a1-b48c-8889059668b2\") "
Jul 28 12:57:53.768503 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03594e8-222e-42e9-b5db-743a9a30ee09-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.768503 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.768503 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.768503 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.768503 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv45p\" (UniqueName: \"kubernetes.io/projected/cc8f1f6e-6b69-4401-b848-f13449658243-kube-api-access-dv45p\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.768782 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a03594e8-222e-42e9-b5db-743a9a30ee09-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.768782 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768579    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f5131c5-4d1a-40a1-b48c-8889059668b2" (UID: "9f5131c5-4d1a-40a1-b48c-8889059668b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:53.768782 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768583    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f5131c5-4d1a-40a1-b48c-8889059668b2" (UID: "9f5131c5-4d1a-40a1-b48c-8889059668b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:53.768782 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-cabundle-cert\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.768782 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768655    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5131c5-4d1a-40a1-b48c-8889059668b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f5131c5-4d1a-40a1-b48c-8889059668b2" (UID: "9f5131c5-4d1a-40a1-b48c-8889059668b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:57:53.768782 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc8f1f6e-6b69-4401-b848-f13449658243-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.768782 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkvl\" (UniqueName: \"kubernetes.io/projected/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-api-access-dzkvl\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.768782 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768785    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc8f1f6e-6b69-4401-b848-f13449658243-proxy-tls\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.769115 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768839    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:57:53.769115 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.768996    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f5131c5-4d1a-40a1-b48c-8889059668b2-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:57:53.769115 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.769072    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:57:53.770463 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.770435    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-api-access-tq8m8" (OuterVolumeSpecName: "kube-api-access-tq8m8") pod "9f5131c5-4d1a-40a1-b48c-8889059668b2" (UID: "9f5131c5-4d1a-40a1-b48c-8889059668b2"). InnerVolumeSpecName "kube-api-access-tq8m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:57:53.770714 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.770689    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5131c5-4d1a-40a1-b48c-8889059668b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f5131c5-4d1a-40a1-b48c-8889059668b2" (UID: "9f5131c5-4d1a-40a1-b48c-8889059668b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:57:53.869794 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.869675    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plmjq\" (UniqueName: \"kubernetes.io/projected/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-api-access-plmjq\") pod \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") "
Jul 28 12:57:53.869794 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.869757    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-cabundle-cert\") pod \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") "
Jul 28 12:57:53.869989 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.869816    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kserve-provision-location\") pod \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") "
Jul 28 12:57:53.869989 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.869875    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d75f4d5d-66c3-435b-9dbd-5cd464123c42-proxy-tls\") pod \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") "
Jul 28 12:57:53.869989 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.869898    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-rbac-proxy-sar-config\") pod \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\" (UID: \"d75f4d5d-66c3-435b-9dbd-5cd464123c42\") "
Jul 28 12:57:53.870142 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc8f1f6e-6b69-4401-b848-f13449658243-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.870142 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkvl\" (UniqueName: \"kubernetes.io/projected/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-api-access-dzkvl\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.870142 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc8f1f6e-6b69-4401-b848-f13449658243-proxy-tls\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.870303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03594e8-222e-42e9-b5db-743a9a30ee09-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.870303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.870303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.870303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870161    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d75f4d5d-66c3-435b-9dbd-5cd464123c42" (UID: "d75f4d5d-66c3-435b-9dbd-5cd464123c42"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:57:53.870303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.870303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d75f4d5d-66c3-435b-9dbd-5cd464123c42" (UID: "d75f4d5d-66c3-435b-9dbd-5cd464123c42"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:53.870303 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv45p\" (UniqueName: \"kubernetes.io/projected/cc8f1f6e-6b69-4401-b848-f13449658243-kube-api-access-dv45p\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.870770 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870325    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d75f4d5d-66c3-435b-9dbd-5cd464123c42" (UID: "d75f4d5d-66c3-435b-9dbd-5cd464123c42"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:53.870770 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a03594e8-222e-42e9-b5db-743a9a30ee09-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.870770 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-cabundle-cert\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.870770 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870417    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq8m8\" (UniqueName: \"kubernetes.io/projected/9f5131c5-4d1a-40a1-b48c-8889059668b2-kube-api-access-tq8m8\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:57:53.870770 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870433    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:57:53.870770 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870446    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:57:53.870770 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870460    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f5131c5-4d1a-40a1-b48c-8889059668b2-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:57:53.870770 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.870473    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d75f4d5d-66c3-435b-9dbd-5cd464123c42-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:57:53.871584 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.871204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.871584 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.871238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a03594e8-222e-42e9-b5db-743a9a30ee09-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.871584 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:57:53.871314    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 12:57:53.871584 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:57:53.871379    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a03594e8-222e-42e9-b5db-743a9a30ee09-proxy-tls podName:a03594e8-222e-42e9-b5db-743a9a30ee09 nodeName:}" failed. No retries permitted until 2026-07-28 12:57:54.371357651 +0000 UTC m=+1461.730962234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a03594e8-222e-42e9-b5db-743a9a30ee09-proxy-tls") pod "isvc-canary-promote-v2-predictor-f59b85f46-knb2w" (UID: "a03594e8-222e-42e9-b5db-743a9a30ee09") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 12:57:53.871584 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.871422    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.871584 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.871489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc8f1f6e-6b69-4401-b848-f13449658243-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.872034 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.871888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.872298 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.872278    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-cabundle-cert\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.872450 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.872427    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d75f4d5d-66c3-435b-9dbd-5cd464123c42-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d75f4d5d-66c3-435b-9dbd-5cd464123c42" (UID: "d75f4d5d-66c3-435b-9dbd-5cd464123c42"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:57:53.872496 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.872431    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-api-access-plmjq" (OuterVolumeSpecName: "kube-api-access-plmjq") pod "d75f4d5d-66c3-435b-9dbd-5cd464123c42" (UID: "d75f4d5d-66c3-435b-9dbd-5cd464123c42"). InnerVolumeSpecName "kube-api-access-plmjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:57:53.872865 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.872848    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc8f1f6e-6b69-4401-b848-f13449658243-proxy-tls\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.879245 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.879214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv45p\" (UniqueName: \"kubernetes.io/projected/cc8f1f6e-6b69-4401-b848-f13449658243-kube-api-access-dv45p\") pod \"isvc-canary-promote-predictor-7bcdb47df7-45qb2\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.880446 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.880425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkvl\" (UniqueName: \"kubernetes.io/projected/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-api-access-dzkvl\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:53.944208 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.944166    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:57:53.964558 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.964528    2561 generic.go:356] "Generic (PLEG): container finished" podID="9f5131c5-4d1a-40a1-b48c-8889059668b2" containerID="c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4" exitCode=0
Jul 28 12:57:53.964703 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.964599    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"
Jul 28 12:57:53.964703 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.964599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" event={"ID":"9f5131c5-4d1a-40a1-b48c-8889059668b2","Type":"ContainerDied","Data":"c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4"}
Jul 28 12:57:53.964852 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.964789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54" event={"ID":"9f5131c5-4d1a-40a1-b48c-8889059668b2","Type":"ContainerDied","Data":"b73812570311f6e72c44d82f468a71ccd9bdc83bdd1a1f793211027c89f1d3eb"}
Jul 28 12:57:53.964852 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.964828    2561 scope.go:117] "RemoveContainer" containerID="e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700"
Jul 28 12:57:53.967250 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.967226    2561 generic.go:356] "Generic (PLEG): container finished" podID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerID="5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703" exitCode=0
Jul 28 12:57:53.967378 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.967263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" event={"ID":"d75f4d5d-66c3-435b-9dbd-5cd464123c42","Type":"ContainerDied","Data":"5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703"}
Jul 28 12:57:53.967378 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.967316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd" event={"ID":"d75f4d5d-66c3-435b-9dbd-5cd464123c42","Type":"ContainerDied","Data":"1fb8fccda44271523ec1e7fbcfab1eeca221a4a724a4f323724192c0f947df06"}
Jul 28 12:57:53.967378 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.967326    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"
Jul 28 12:57:53.971056 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.971033    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d75f4d5d-66c3-435b-9dbd-5cd464123c42-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:57:53.971141 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.971062    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plmjq\" (UniqueName: \"kubernetes.io/projected/d75f4d5d-66c3-435b-9dbd-5cd464123c42-kube-api-access-plmjq\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:57:53.973953 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.973922    2561 scope.go:117] "RemoveContainer" containerID="c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4"
Jul 28 12:57:53.983062 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.983045    2561 scope.go:117] "RemoveContainer" containerID="58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7"
Jul 28 12:57:53.990685 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.990661    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"]
Jul 28 12:57:53.991627 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.991610    2561 scope.go:117] "RemoveContainer" containerID="e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700"
Jul 28 12:57:53.991961 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:57:53.991923    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700\": container with ID starting with e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700 not found: ID does not exist" containerID="e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700"
Jul 28 12:57:53.992049 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.991960    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700"} err="failed to get container status \"e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700\": rpc error: code = NotFound desc = could not find container \"e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700\": container with ID starting with e8469b1d6871e3a95038ef5f97b632b076cab79d8bc10cdc8defbd30a90d5700 not found: ID does not exist"
Jul 28 12:57:53.992049 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.991984    2561 scope.go:117] "RemoveContainer" containerID="c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4"
Jul 28 12:57:53.992275 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:57:53.992258    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4\": container with ID starting with c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4 not found: ID does not exist" containerID="c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4"
Jul 28 12:57:53.992340 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.992279    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4"} err="failed to get container status \"c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4\": rpc error: code = NotFound desc = could not find container \"c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4\": container with ID starting with c8d99786076c2500daf9a8702350cbf38c08daa84fbfd29a8eb98161f450d1a4 not found: ID does not exist"
Jul 28 12:57:53.992340 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.992293    2561 scope.go:117] "RemoveContainer" containerID="58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7"
Jul 28 12:57:53.992567 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:57:53.992550    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7\": container with ID starting with 58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7 not found: ID does not exist" containerID="58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7"
Jul 28 12:57:53.992636 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.992571    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7"} err="failed to get container status \"58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7\": rpc error: code = NotFound desc = could not find container \"58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7\": container with ID starting with 58d19865c3f79cbab1e6cabaecd6bb3549843ca703dd3c53029bdf0b343cdcf7 not found: ID does not exist"
Jul 28 12:57:53.992636 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.992585    2561 scope.go:117] "RemoveContainer" containerID="8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6"
Jul 28 12:57:53.995908 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:53.995884    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-bss54"]
Jul 28 12:57:54.002291 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.002250    2561 scope.go:117] "RemoveContainer" containerID="5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703"
Jul 28 12:57:54.006777 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.006639    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:54.007649 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.007621    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"]
Jul 28 12:57:54.010007 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.009606    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-lcbsd"]
Jul 28 12:57:54.012656 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.012636    2561 scope.go:117] "RemoveContainer" containerID="332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c"
Jul 28 12:57:54.022406 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.022385    2561 scope.go:117] "RemoveContainer" containerID="8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6"
Jul 28 12:57:54.022721 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:57:54.022698    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6\": container with ID starting with 8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6 not found: ID does not exist" containerID="8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6"
Jul 28 12:57:54.022844 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.022753    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6"} err="failed to get container status \"8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6\": rpc error: code = NotFound desc = could not find container \"8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6\": container with ID starting with 8d0aee48ba812faf366f500438d6c4e2c8b41024496480cad1f6da91343ee3d6 not found: ID does not exist"
Jul 28 12:57:54.022844 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.022781    2561 scope.go:117] "RemoveContainer" containerID="5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703"
Jul 28 12:57:54.023084 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:57:54.023058    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703\": container with ID starting with 5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703 not found: ID does not exist" containerID="5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703"
Jul 28 12:57:54.023141 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.023090    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703"} err="failed to get container status \"5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703\": rpc error: code = NotFound desc = could not find container \"5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703\": container with ID starting with 5598ee3a6b29381314e28a93f7c5d3e539ca6972c27def6c0d413f1e09c34703 not found: ID does not exist"
Jul 28 12:57:54.023141 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.023109    2561 scope.go:117] "RemoveContainer" containerID="332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c"
Jul 28 12:57:54.023406 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:57:54.023387    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c\": container with ID starting with 332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c not found: ID does not exist" containerID="332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c"
Jul 28 12:57:54.023502 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.023408    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c"} err="failed to get container status \"332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c\": rpc error: code = NotFound desc = could not find container \"332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c\": container with ID starting with 332ed29a20bb2f219eed468173a03d9201ca446609c6f578f7d645322804ed2c not found: ID does not exist"
Jul 28 12:57:54.086360 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.086336    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"]
Jul 28 12:57:54.088671 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:57:54.088645    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8f1f6e_6b69_4401_b848_f13449658243.slice/crio-af0a015c2dcc9a0783288c84aa680ea2c372b72adcacd481ff33766b4e2876e5 WatchSource:0}: Error finding container af0a015c2dcc9a0783288c84aa680ea2c372b72adcacd481ff33766b4e2876e5: Status 404 returned error can't find the container with id af0a015c2dcc9a0783288c84aa680ea2c372b72adcacd481ff33766b4e2876e5
Jul 28 12:57:54.374803 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.374705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03594e8-222e-42e9-b5db-743a9a30ee09-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:54.377221 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.377196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03594e8-222e-42e9-b5db-743a9a30ee09-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-knb2w\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:54.534094 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.534056    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:57:54.670863 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.670833    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"]
Jul 28 12:57:54.673006 ip-10-0-138-58 kubenswrapper[2561]: W0728 12:57:54.672978    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03594e8_222e_42e9_b5db_743a9a30ee09.slice/crio-36e6ef343ecb44927b8c77ceca01e5bd255dcb6bde34771ffc663b727378671e WatchSource:0}: Error finding container 36e6ef343ecb44927b8c77ceca01e5bd255dcb6bde34771ffc663b727378671e: Status 404 returned error can't find the container with id 36e6ef343ecb44927b8c77ceca01e5bd255dcb6bde34771ffc663b727378671e
Jul 28 12:57:54.974243 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.974155    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" event={"ID":"a03594e8-222e-42e9-b5db-743a9a30ee09","Type":"ContainerStarted","Data":"2f7949d99af1218c60bf564d24a615abce0d8c201dcf7f0981c178c6cd76c176"}
Jul 28 12:57:54.974243 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.974226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" event={"ID":"a03594e8-222e-42e9-b5db-743a9a30ee09","Type":"ContainerStarted","Data":"36e6ef343ecb44927b8c77ceca01e5bd255dcb6bde34771ffc663b727378671e"}
Jul 28 12:57:54.975697 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.975673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" event={"ID":"cc8f1f6e-6b69-4401-b848-f13449658243","Type":"ContainerStarted","Data":"f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6"}
Jul 28 12:57:54.975829 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:54.975759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" event={"ID":"cc8f1f6e-6b69-4401-b848-f13449658243","Type":"ContainerStarted","Data":"af0a015c2dcc9a0783288c84aa680ea2c372b72adcacd481ff33766b4e2876e5"}
Jul 28 12:57:55.102140 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:55.102107    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f5131c5-4d1a-40a1-b48c-8889059668b2" path="/var/lib/kubelet/pods/9f5131c5-4d1a-40a1-b48c-8889059668b2/volumes"
Jul 28 12:57:55.102587 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:55.102570    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" path="/var/lib/kubelet/pods/d75f4d5d-66c3-435b-9dbd-5cd464123c42/volumes"
Jul 28 12:57:58.995825 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:58.995785    2561 generic.go:356] "Generic (PLEG): container finished" podID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerID="2f7949d99af1218c60bf564d24a615abce0d8c201dcf7f0981c178c6cd76c176" exitCode=0
Jul 28 12:57:58.996289 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:58.995859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" event={"ID":"a03594e8-222e-42e9-b5db-743a9a30ee09","Type":"ContainerDied","Data":"2f7949d99af1218c60bf564d24a615abce0d8c201dcf7f0981c178c6cd76c176"}
Jul 28 12:57:58.997525 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:58.997504    2561 generic.go:356] "Generic (PLEG): container finished" podID="cc8f1f6e-6b69-4401-b848-f13449658243" containerID="f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6" exitCode=0
Jul 28 12:57:58.997629 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:57:58.997544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" event={"ID":"cc8f1f6e-6b69-4401-b848-f13449658243","Type":"ContainerDied","Data":"f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6"}
Jul 28 12:58:00.003907 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.003870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" event={"ID":"a03594e8-222e-42e9-b5db-743a9a30ee09","Type":"ContainerStarted","Data":"45f20967a1006b2fdf54776949973baf2058c354eeb7c079179cb285d8455343"}
Jul 28 12:58:00.004398 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.003949    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" event={"ID":"a03594e8-222e-42e9-b5db-743a9a30ee09","Type":"ContainerStarted","Data":"d0f9182297ea635599c36cf8fed288ec2e8af6c17a5357d037f250a0227f9294"}
Jul 28 12:58:00.004398 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.004261    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:58:00.004520 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.004414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:58:00.005884 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.005857    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 12:58:00.006008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.005958    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" event={"ID":"cc8f1f6e-6b69-4401-b848-f13449658243","Type":"ContainerStarted","Data":"043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af"}
Jul 28 12:58:00.006008 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.006002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" event={"ID":"cc8f1f6e-6b69-4401-b848-f13449658243","Type":"ContainerStarted","Data":"acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac"}
Jul 28 12:58:00.006211 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.006195    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:58:00.006284 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.006245    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:58:00.007276 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.007251    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 12:58:00.024591 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.024538    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podStartSLOduration=3.258623119 podStartE2EDuration="7.024511049s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.76588793s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:58:00.022752892 +0000 UTC m=+1467.382357488" watchObservedRunningTime="2026-07-28 12:58:00.024511049 +0000 UTC m=+1467.384115654"
Jul 28 12:58:00.042971 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:00.042913    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podStartSLOduration=3.19407139 podStartE2EDuration="7.042873878s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.848802488s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:58:00.042144896 +0000 UTC m=+1467.401749503" watchObservedRunningTime="2026-07-28 12:58:00.042873878 +0000 UTC m=+1467.402478482"
Jul 28 12:58:01.010586 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:01.010543    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 12:58:01.011093 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:01.010764    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 12:58:06.015761 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:06.015716    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:58:06.016113 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:06.015835    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:58:06.016352 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:06.016324    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 12:58:06.016661 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:06.016637    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 12:58:16.016978 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:16.016935    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 12:58:16.017358 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:16.016935    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 12:58:26.016871 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:26.016833    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 12:58:26.017334 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:26.016849    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 12:58:33.093212 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:33.093186    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:58:33.095399 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:33.095377    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 12:58:36.017014 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:36.016975    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 12:58:36.017401 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:36.016976    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 12:58:46.017345 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:46.017303    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 12:58:46.017915 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:46.017304    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 12:58:46.987197 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:46.987162    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:58:56.016862 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:56.016768    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 12:58:56.017312 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:56.016775    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 12:58:57.841287 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:58:57.841245    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:06.016404 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:06.016364    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 12:59:06.016943 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:06.016632    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 12:59:16.017394 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:16.017361    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:59:16.017886 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:16.017454    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 12:59:20.238646 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:20.238608    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:23.711339 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:23.711295    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"]
Jul 28 12:59:23.711889 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:23.711544    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" containerID="cri-o://acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac" gracePeriod=30
Jul 28 12:59:23.711889 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:23.711588    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kube-rbac-proxy" containerID="cri-o://043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af" gracePeriod=30
Jul 28 12:59:23.731389 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:23.731359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:24.345367 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:24.345334    2561 generic.go:356] "Generic (PLEG): container finished" podID="cc8f1f6e-6b69-4401-b848-f13449658243" containerID="043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af" exitCode=2
Jul 28 12:59:24.345552 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:24.345383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" event={"ID":"cc8f1f6e-6b69-4401-b848-f13449658243","Type":"ContainerDied","Data":"043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af"}
Jul 28 12:59:26.011339 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:26.011288    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 12:59:26.016393 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:26.016363    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 12:59:28.258062 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.258033    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:28.661493 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.661467    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:59:28.780567 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.780539    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-kube-rbac-proxy-sar-config\") pod \"cc8f1f6e-6b69-4401-b848-f13449658243\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") "
Jul 28 12:59:28.780767 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.780578    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc8f1f6e-6b69-4401-b848-f13449658243-proxy-tls\") pod \"cc8f1f6e-6b69-4401-b848-f13449658243\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") "
Jul 28 12:59:28.780767 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.780600    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-cabundle-cert\") pod \"cc8f1f6e-6b69-4401-b848-f13449658243\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") "
Jul 28 12:59:28.780767 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.780644    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc8f1f6e-6b69-4401-b848-f13449658243-kserve-provision-location\") pod \"cc8f1f6e-6b69-4401-b848-f13449658243\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") "
Jul 28 12:59:28.780767 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.780683    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv45p\" (UniqueName: \"kubernetes.io/projected/cc8f1f6e-6b69-4401-b848-f13449658243-kube-api-access-dv45p\") pod \"cc8f1f6e-6b69-4401-b848-f13449658243\" (UID: \"cc8f1f6e-6b69-4401-b848-f13449658243\") "
Jul 28 12:59:28.781060 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.781022    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8f1f6e-6b69-4401-b848-f13449658243-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc8f1f6e-6b69-4401-b848-f13449658243" (UID: "cc8f1f6e-6b69-4401-b848-f13449658243"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:59:28.781060 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.781033    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc8f1f6e-6b69-4401-b848-f13449658243" (UID: "cc8f1f6e-6b69-4401-b848-f13449658243"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:59:28.781165 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.781039    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc8f1f6e-6b69-4401-b848-f13449658243" (UID: "cc8f1f6e-6b69-4401-b848-f13449658243"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:59:28.782901 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.782882    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8f1f6e-6b69-4401-b848-f13449658243-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc8f1f6e-6b69-4401-b848-f13449658243" (UID: "cc8f1f6e-6b69-4401-b848-f13449658243"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:59:28.782990 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.782962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8f1f6e-6b69-4401-b848-f13449658243-kube-api-access-dv45p" (OuterVolumeSpecName: "kube-api-access-dv45p") pod "cc8f1f6e-6b69-4401-b848-f13449658243" (UID: "cc8f1f6e-6b69-4401-b848-f13449658243"). InnerVolumeSpecName "kube-api-access-dv45p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:59:28.881970 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.881916    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv45p\" (UniqueName: \"kubernetes.io/projected/cc8f1f6e-6b69-4401-b848-f13449658243-kube-api-access-dv45p\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:59:28.881970 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.881960    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:59:28.881970 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.881970    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc8f1f6e-6b69-4401-b848-f13449658243-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:59:28.881970 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.881982    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc8f1f6e-6b69-4401-b848-f13449658243-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:59:28.881970 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:28.881990    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc8f1f6e-6b69-4401-b848-f13449658243-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 12:59:29.366803 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.366689    2561 generic.go:356] "Generic (PLEG): container finished" podID="cc8f1f6e-6b69-4401-b848-f13449658243" containerID="acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac" exitCode=0
Jul 28 12:59:29.366803 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.366763    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" event={"ID":"cc8f1f6e-6b69-4401-b848-f13449658243","Type":"ContainerDied","Data":"acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac"}
Jul 28 12:59:29.366803 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.366800    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"
Jul 28 12:59:29.367296 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.366828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2" event={"ID":"cc8f1f6e-6b69-4401-b848-f13449658243","Type":"ContainerDied","Data":"af0a015c2dcc9a0783288c84aa680ea2c372b72adcacd481ff33766b4e2876e5"}
Jul 28 12:59:29.367296 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.366867    2561 scope.go:117] "RemoveContainer" containerID="043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af"
Jul 28 12:59:29.375234 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.375212    2561 scope.go:117] "RemoveContainer" containerID="acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac"
Jul 28 12:59:29.382660 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.382640    2561 scope.go:117] "RemoveContainer" containerID="f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6"
Jul 28 12:59:29.385673 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.385643    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"]
Jul 28 12:59:29.390318 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.390296    2561 scope.go:117] "RemoveContainer" containerID="043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af"
Jul 28 12:59:29.390758 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:59:29.390713    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af\": container with ID starting with 043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af not found: ID does not exist" containerID="043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af"
Jul 28 12:59:29.391020 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.390770    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af"} err="failed to get container status \"043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af\": rpc error: code = NotFound desc = could not find container \"043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af\": container with ID starting with 043a45036e196dd47fa34e4daae502d3c1c88c871a86108d01aa60a3cc8132af not found: ID does not exist"
Jul 28 12:59:29.391020 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.390797    2561 scope.go:117] "RemoveContainer" containerID="acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac"
Jul 28 12:59:29.391115 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:59:29.391076    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac\": container with ID starting with acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac not found: ID does not exist" containerID="acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac"
Jul 28 12:59:29.391155 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.391105    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac"} err="failed to get container status \"acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac\": rpc error: code = NotFound desc = could not find container \"acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac\": container with ID starting with acf33ba146e9e2627ec5e6bbb840c9f2705b9f996bd51e5009a4f5b2324fbdac not found: ID does not exist"
Jul 28 12:59:29.391155 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.391127    2561 scope.go:117] "RemoveContainer" containerID="f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6"
Jul 28 12:59:29.391409 ip-10-0-138-58 kubenswrapper[2561]: E0728 12:59:29.391392    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6\": container with ID starting with f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6 not found: ID does not exist" containerID="f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6"
Jul 28 12:59:29.391460 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.391414    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6"} err="failed to get container status \"f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6\": rpc error: code = NotFound desc = could not find container \"f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6\": container with ID starting with f147f04a260591cc338a6cce5bf9add544ea7672020784962c45e761c05471f6 not found: ID does not exist"
Jul 28 12:59:29.392249 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:29.392229    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-45qb2"]
Jul 28 12:59:31.102301 ip-10-0-138-58 kubenswrapper[2561]: I0728 12:59:31.102267    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" path="/var/lib/kubelet/pods/cc8f1f6e-6b69-4401-b848-f13449658243/volumes"
Jul 28 13:01:03.975529 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:03.975489    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"]
Jul 28 13:01:03.976146 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:03.975827    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" containerID="cri-o://d0f9182297ea635599c36cf8fed288ec2e8af6c17a5357d037f250a0227f9294" gracePeriod=30
Jul 28 13:01:03.976146 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:03.975907    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kube-rbac-proxy" containerID="cri-o://45f20967a1006b2fdf54776949973baf2058c354eeb7c079179cb285d8455343" gracePeriod=30
Jul 28 13:01:04.029180 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:04.029146    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:04.740027 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:04.739994    2561 generic.go:356] "Generic (PLEG): container finished" podID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerID="45f20967a1006b2fdf54776949973baf2058c354eeb7c079179cb285d8455343" exitCode=2
Jul 28 13:01:04.740205 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:04.740074    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" event={"ID":"a03594e8-222e-42e9-b5db-743a9a30ee09","Type":"ContainerDied","Data":"45f20967a1006b2fdf54776949973baf2058c354eeb7c079179cb285d8455343"}
Jul 28 13:01:06.011521 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:06.011479    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 28 13:01:06.016926 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:06.016903    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 13:01:08.757188 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.757153    2561 generic.go:356] "Generic (PLEG): container finished" podID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerID="d0f9182297ea635599c36cf8fed288ec2e8af6c17a5357d037f250a0227f9294" exitCode=0
Jul 28 13:01:08.757522 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.757217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" event={"ID":"a03594e8-222e-42e9-b5db-743a9a30ee09","Type":"ContainerDied","Data":"d0f9182297ea635599c36cf8fed288ec2e8af6c17a5357d037f250a0227f9294"}
Jul 28 13:01:08.827583 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.827560    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 13:01:08.918002 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.917909    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-cabundle-cert\") pod \"a03594e8-222e-42e9-b5db-743a9a30ee09\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") "
Jul 28 13:01:08.918002 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.917965    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03594e8-222e-42e9-b5db-743a9a30ee09-proxy-tls\") pod \"a03594e8-222e-42e9-b5db-743a9a30ee09\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") "
Jul 28 13:01:08.918002 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.917998    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a03594e8-222e-42e9-b5db-743a9a30ee09-kserve-provision-location\") pod \"a03594e8-222e-42e9-b5db-743a9a30ee09\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") "
Jul 28 13:01:08.918266 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.918027    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-rbac-proxy-sar-config\") pod \"a03594e8-222e-42e9-b5db-743a9a30ee09\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") "
Jul 28 13:01:08.918266 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.918060    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzkvl\" (UniqueName: \"kubernetes.io/projected/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-api-access-dzkvl\") pod \"a03594e8-222e-42e9-b5db-743a9a30ee09\" (UID: \"a03594e8-222e-42e9-b5db-743a9a30ee09\") "
Jul 28 13:01:08.918377 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.918326    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a03594e8-222e-42e9-b5db-743a9a30ee09" (UID: "a03594e8-222e-42e9-b5db-743a9a30ee09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:08.918377 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.918331    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03594e8-222e-42e9-b5db-743a9a30ee09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a03594e8-222e-42e9-b5db-743a9a30ee09" (UID: "a03594e8-222e-42e9-b5db-743a9a30ee09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:01:08.918487 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.918387    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a03594e8-222e-42e9-b5db-743a9a30ee09" (UID: "a03594e8-222e-42e9-b5db-743a9a30ee09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:08.920175 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.920158    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03594e8-222e-42e9-b5db-743a9a30ee09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a03594e8-222e-42e9-b5db-743a9a30ee09" (UID: "a03594e8-222e-42e9-b5db-743a9a30ee09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:01:08.920389 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:08.920368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-api-access-dzkvl" (OuterVolumeSpecName: "kube-api-access-dzkvl") pod "a03594e8-222e-42e9-b5db-743a9a30ee09" (UID: "a03594e8-222e-42e9-b5db-743a9a30ee09"). InnerVolumeSpecName "kube-api-access-dzkvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:01:09.019317 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.019271    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkvl\" (UniqueName: \"kubernetes.io/projected/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-api-access-dzkvl\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:01:09.019317 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.019311    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:01:09.019317 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.019327    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a03594e8-222e-42e9-b5db-743a9a30ee09-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:01:09.019567 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.019341    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a03594e8-222e-42e9-b5db-743a9a30ee09-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:01:09.019567 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.019353    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a03594e8-222e-42e9-b5db-743a9a30ee09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:01:09.076660 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.076624    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:09.171633 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.171549    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:09.283399 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.283363    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:09.763080 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.763046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w" event={"ID":"a03594e8-222e-42e9-b5db-743a9a30ee09","Type":"ContainerDied","Data":"36e6ef343ecb44927b8c77ceca01e5bd255dcb6bde34771ffc663b727378671e"}
Jul 28 13:01:09.763080 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.763079    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"
Jul 28 13:01:09.763614 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.763112    2561 scope.go:117] "RemoveContainer" containerID="45f20967a1006b2fdf54776949973baf2058c354eeb7c079179cb285d8455343"
Jul 28 13:01:09.771400 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.771380    2561 scope.go:117] "RemoveContainer" containerID="d0f9182297ea635599c36cf8fed288ec2e8af6c17a5357d037f250a0227f9294"
Jul 28 13:01:09.779688 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.779671    2561 scope.go:117] "RemoveContainer" containerID="2f7949d99af1218c60bf564d24a615abce0d8c201dcf7f0981c178c6cd76c176"
Jul 28 13:01:09.790485 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.790463    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"]
Jul 28 13:01:09.794503 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.794482    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-knb2w"]
Jul 28 13:01:09.909534 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:09.909499    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:11.101712 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:01:11.101676    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" path="/var/lib/kubelet/pods/a03594e8-222e-42e9-b5db-743a9a30ee09/volumes"
Jul 28 13:02:02.025938 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:02.025845    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:02:39.199319 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.199267    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:02:39.249413 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249379    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"]
Jul 28 13:02:39.249659 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249648    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container"
Jul 28 13:02:39.249702 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249661    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container"
Jul 28 13:02:39.249702 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249675    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kube-rbac-proxy"
Jul 28 13:02:39.249702 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249683    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kube-rbac-proxy"
Jul 28 13:02:39.249702 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249695    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249703    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249713    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kube-rbac-proxy"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249719    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kube-rbac-proxy"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249740    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="storage-initializer"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249746    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="storage-initializer"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249753    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249758    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249767    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="storage-initializer"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249775    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="storage-initializer"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249788    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kube-rbac-proxy"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249795    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kube-rbac-proxy"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249803    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="storage-initializer"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249808    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="storage-initializer"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249856    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kserve-container"
Jul 28 13:02:39.249860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249864    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kube-rbac-proxy"
Jul 28 13:02:39.250290 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249871    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kserve-container"
Jul 28 13:02:39.250290 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249877    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a03594e8-222e-42e9-b5db-743a9a30ee09" containerName="kube-rbac-proxy"
Jul 28 13:02:39.250290 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249884    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d75f4d5d-66c3-435b-9dbd-5cd464123c42" containerName="kserve-container"
Jul 28 13:02:39.250290 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.249889    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8f1f6e-6b69-4401-b848-f13449658243" containerName="kube-rbac-proxy"
Jul 28 13:02:39.252795 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.252779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.255415 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.255384    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 13:02:39.255555 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.255428    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 13:02:39.255555 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.255456    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:02:39.256042 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.256028    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:02:39.256346 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.256334    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rtw6d\""
Jul 28 13:02:39.256390 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.256381    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:02:39.256423 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.256395    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:02:39.263956 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.263935    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"]
Jul 28 13:02:39.314195 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.314163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.314370 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.314206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.314370 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.314237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.314370 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.314297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.314370 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.314331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9czh\" (UniqueName: \"kubernetes.io/projected/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-api-access-z9czh\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.415267 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.415224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.415468 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.415285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.415468 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.415324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.415468 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.415355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9czh\" (UniqueName: \"kubernetes.io/projected/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-api-access-z9czh\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.415468 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.415376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.415669 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.415649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.416095 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.416042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.416095 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.416050    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.417887 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.417867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.423103 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.423081    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9czh\" (UniqueName: \"kubernetes.io/projected/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-api-access-z9czh\") pod \"isvc-canary-rollback-predictor-ff46f797c-5qthq\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.563508 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.563472    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:39.699746 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:39.699701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"]
Jul 28 13:02:39.701593 ip-10-0-138-58 kubenswrapper[2561]: W0728 13:02:39.701567    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623beffe_1aaa_4dae_8be3_d3d2ea9e5a6c.slice/crio-c95925df9dfe53777dc3b2035e7cb656ad71d0b37f4e840fefbff287560469ad WatchSource:0}: Error finding container c95925df9dfe53777dc3b2035e7cb656ad71d0b37f4e840fefbff287560469ad: Status 404 returned error can't find the container with id c95925df9dfe53777dc3b2035e7cb656ad71d0b37f4e840fefbff287560469ad
Jul 28 13:02:40.101896 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:40.101859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" event={"ID":"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c","Type":"ContainerStarted","Data":"5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835"}
Jul 28 13:02:40.102055 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:40.101916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" event={"ID":"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c","Type":"ContainerStarted","Data":"c95925df9dfe53777dc3b2035e7cb656ad71d0b37f4e840fefbff287560469ad"}
Jul 28 13:02:44.117646 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:44.117611    2561 generic.go:356] "Generic (PLEG): container finished" podID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerID="5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835" exitCode=0
Jul 28 13:02:44.118034 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:44.117685    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" event={"ID":"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c","Type":"ContainerDied","Data":"5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835"}
Jul 28 13:02:45.122892 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:45.122858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" event={"ID":"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c","Type":"ContainerStarted","Data":"b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687"}
Jul 28 13:02:45.123340 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:45.122912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" event={"ID":"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c","Type":"ContainerStarted","Data":"0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28"}
Jul 28 13:02:45.123340 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:45.123164    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:45.123340 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:45.123297    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:45.124497 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:45.124475    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:02:45.143103 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:45.143053    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podStartSLOduration=2.170104435 podStartE2EDuration="6.143040053s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.972935618s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:02:45.14142603 +0000 UTC m=+1752.501030638" watchObservedRunningTime="2026-07-28 13:02:45.143040053 +0000 UTC m=+1752.502645061"
Jul 28 13:02:46.126536 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:46.126496    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:02:51.130954 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:51.130928    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:02:51.131461 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:02:51.131434    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:03:01.131572 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:03:01.131533    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:03:11.131621 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:03:11.131580    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:03:21.132084 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:03:21.132043    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:03:31.131957 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:03:31.131917    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:03:33.119472 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:03:33.119438    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 13:03:33.119902 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:03:33.119518    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 13:03:41.132191 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:03:41.132150    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:03:51.131684 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:03:51.131644    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:04:01.132636 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:01.132603    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:04:09.385038 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:09.385006    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"]
Jul 28 13:04:09.385474 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:09.385293    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" containerID="cri-o://0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28" gracePeriod=30
Jul 28 13:04:09.385474 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:09.385330    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kube-rbac-proxy" containerID="cri-o://b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687" gracePeriod=30
Jul 28 13:04:09.459571 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:09.459533    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:04:10.455302 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:10.455258    2561 generic.go:356] "Generic (PLEG): container finished" podID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerID="b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687" exitCode=2
Jul 28 13:04:10.455686 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:10.455328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" event={"ID":"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c","Type":"ContainerDied","Data":"b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687"}
Jul 28 13:04:11.127525 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:11.127484    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 28 13:04:11.131826 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:11.131796    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 13:04:14.337544 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.337519    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:04:14.415087 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.415006    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kserve-provision-location\") pod \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") "
Jul 28 13:04:14.415087 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.415069    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9czh\" (UniqueName: \"kubernetes.io/projected/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-api-access-z9czh\") pod \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") "
Jul 28 13:04:14.415087 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.415090    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-proxy-tls\") pod \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") "
Jul 28 13:04:14.415403 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.415123    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-rbac-proxy-sar-config\") pod \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") "
Jul 28 13:04:14.415403 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.415156    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-cabundle-cert\") pod \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\" (UID: \"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c\") "
Jul 28 13:04:14.415487 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.415421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" (UID: "623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:04:14.415522 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.415499    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" (UID: "623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:04:14.415587 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.415558    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" (UID: "623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:04:14.417336 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.417319    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" (UID: "623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:04:14.417415 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.417397    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-api-access-z9czh" (OuterVolumeSpecName: "kube-api-access-z9czh") pod "623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" (UID: "623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c"). InnerVolumeSpecName "kube-api-access-z9czh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:04:14.477586 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.477548    2561 generic.go:356] "Generic (PLEG): container finished" podID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerID="0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28" exitCode=0
Jul 28 13:04:14.477761 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.477624    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"
Jul 28 13:04:14.477761 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.477630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" event={"ID":"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c","Type":"ContainerDied","Data":"0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28"}
Jul 28 13:04:14.477761 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.477701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq" event={"ID":"623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c","Type":"ContainerDied","Data":"c95925df9dfe53777dc3b2035e7cb656ad71d0b37f4e840fefbff287560469ad"}
Jul 28 13:04:14.477877 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.477773    2561 scope.go:117] "RemoveContainer" containerID="b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687"
Jul 28 13:04:14.489475 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.489456    2561 scope.go:117] "RemoveContainer" containerID="0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28"
Jul 28 13:04:14.492719 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.492695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:04:14.499925 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.499904    2561 scope.go:117] "RemoveContainer" containerID="5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835"
Jul 28 13:04:14.505869 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.505841    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"]
Jul 28 13:04:14.508603 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.508584    2561 scope.go:117] "RemoveContainer" containerID="b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687"
Jul 28 13:04:14.509023 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:04:14.508996    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687\": container with ID starting with b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687 not found: ID does not exist" containerID="b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687"
Jul 28 13:04:14.509110 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.509036    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687"} err="failed to get container status \"b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687\": rpc error: code = NotFound desc = could not find container \"b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687\": container with ID starting with b2c7017b24384291fc6bd6a845310a2fb891532456b340beb1e770512d7eb687 not found: ID does not exist"
Jul 28 13:04:14.509110 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.509061    2561 scope.go:117] "RemoveContainer" containerID="0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28"
Jul 28 13:04:14.509363 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:04:14.509338    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28\": container with ID starting with 0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28 not found: ID does not exist" containerID="0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28"
Jul 28 13:04:14.509453 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.509372    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28"} err="failed to get container status \"0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28\": rpc error: code = NotFound desc = could not find container \"0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28\": container with ID starting with 0289b444a2b913974771fc76d1d1acdf95de2a5cd48b7d1a85a739af3fcc0a28 not found: ID does not exist"
Jul 28 13:04:14.509453 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.509394    2561 scope.go:117] "RemoveContainer" containerID="5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835"
Jul 28 13:04:14.509799 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:04:14.509775    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835\": container with ID starting with 5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835 not found: ID does not exist" containerID="5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835"
Jul 28 13:04:14.509904 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.509806    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835"} err="failed to get container status \"5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835\": rpc error: code = NotFound desc = could not find container \"5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835\": container with ID starting with 5d7995e31b6cbb85fc075369e6e84ad7327b495f18e9f7d367d05ed6e6971835 not found: ID does not exist"
Jul 28 13:04:14.513217 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.513193    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-5qthq"]
Jul 28 13:04:14.515951 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.515930    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:04:14.516052 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.515956    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:04:14.516052 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.515974    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:04:14.516052 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.515987    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9czh\" (UniqueName: \"kubernetes.io/projected/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-kube-api-access-z9czh\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:04:14.516052 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.515999    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:04:14.563107 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.563032    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:04:14.566436 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.566413    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"]
Jul 28 13:04:14.566709 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.566694    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="storage-initializer"
Jul 28 13:04:14.566767 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.566712    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="storage-initializer"
Jul 28 13:04:14.566767 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.566725    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container"
Jul 28 13:04:14.566767 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.566748    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container"
Jul 28 13:04:14.566767 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.566758    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kube-rbac-proxy"
Jul 28 13:04:14.566767 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.566763    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kube-rbac-proxy"
Jul 28 13:04:14.566922 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.566822    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kserve-container"
Jul 28 13:04:14.566922 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.566831    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" containerName="kube-rbac-proxy"
Jul 28 13:04:14.577335 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.577306    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.579972 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.579947    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:04:14.580102 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.579954    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:04:14.580102 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.579966    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:04:14.580235 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.579973    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 13:04:14.580235 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.579996    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 13:04:14.580235 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.580020    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rtw6d\""
Jul 28 13:04:14.580235 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.580042    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:04:14.583088 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.583066    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"]
Jul 28 13:04:14.616374 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.616339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.616524 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.616383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2wwr\" (UniqueName: \"kubernetes.io/projected/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-api-access-w2wwr\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.616524 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.616416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5beab22d-ca46-4657-bd31-d83cc74ef79e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.616598 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.616530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.616598 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.616585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.717386 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.717289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.717386 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.717342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.717616 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:04:14.717467    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 13:04:14.717616 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.717467    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wwr\" (UniqueName: \"kubernetes.io/projected/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-api-access-w2wwr\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.717616 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:04:14.717547    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls podName:5beab22d-ca46-4657-bd31-d83cc74ef79e nodeName:}" failed. No retries permitted until 2026-07-28 13:04:15.217524414 +0000 UTC m=+1842.577129011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls") pod "isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" (UID: "5beab22d-ca46-4657-bd31-d83cc74ef79e") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 13:04:14.717616 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.717602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5beab22d-ca46-4657-bd31-d83cc74ef79e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.717859 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.717688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.718010 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.717990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.718092 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.718053    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5beab22d-ca46-4657-bd31-d83cc74ef79e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.718342 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.718324    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.727467 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.727439    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wwr\" (UniqueName: \"kubernetes.io/projected/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-api-access-w2wwr\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:14.784875 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:14.784843    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:04:15.102852 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:15.102817    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c" path="/var/lib/kubelet/pods/623beffe-1aaa-4dae-8be3-d3d2ea9e5a6c/volumes"
Jul 28 13:04:15.221117 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:15.221079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:15.221297 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:04:15.221242    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 13:04:15.221339 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:04:15.221318    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls podName:5beab22d-ca46-4657-bd31-d83cc74ef79e nodeName:}" failed. No retries permitted until 2026-07-28 13:04:16.221301994 +0000 UTC m=+1843.580906577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls") pod "isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" (UID: "5beab22d-ca46-4657-bd31-d83cc74ef79e") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 13:04:15.574307 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:15.574271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:04:16.229261 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:16.229224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:16.231863 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:16.231833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:16.390799 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:16.390756    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:16.523989 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:16.523963    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"]
Jul 28 13:04:16.526102 ip-10-0-138-58 kubenswrapper[2561]: W0728 13:04:16.526063    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5beab22d_ca46_4657_bd31_d83cc74ef79e.slice/crio-a32150d905864f40545115edfc2de046a001ebc279e9338e601312fa1be7a984 WatchSource:0}: Error finding container a32150d905864f40545115edfc2de046a001ebc279e9338e601312fa1be7a984: Status 404 returned error can't find the container with id a32150d905864f40545115edfc2de046a001ebc279e9338e601312fa1be7a984
Jul 28 13:04:17.492598 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:17.492558    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" event={"ID":"5beab22d-ca46-4657-bd31-d83cc74ef79e","Type":"ContainerStarted","Data":"952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3"}
Jul 28 13:04:17.493089 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:17.492624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" event={"ID":"5beab22d-ca46-4657-bd31-d83cc74ef79e","Type":"ContainerStarted","Data":"a32150d905864f40545115edfc2de046a001ebc279e9338e601312fa1be7a984"}
Jul 28 13:04:20.506575 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:20.506539    2561 generic.go:356] "Generic (PLEG): container finished" podID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerID="952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3" exitCode=0
Jul 28 13:04:20.506975 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:20.506616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" event={"ID":"5beab22d-ca46-4657-bd31-d83cc74ef79e","Type":"ContainerDied","Data":"952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3"}
Jul 28 13:04:21.512851 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:21.512817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" event={"ID":"5beab22d-ca46-4657-bd31-d83cc74ef79e","Type":"ContainerStarted","Data":"683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf"}
Jul 28 13:04:21.513217 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:21.512878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" event={"ID":"5beab22d-ca46-4657-bd31-d83cc74ef79e","Type":"ContainerStarted","Data":"201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec"}
Jul 28 13:04:21.513217 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:21.513105    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:21.513360 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:21.513253    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:21.514448 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:21.514421    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:04:21.531724 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:21.531678    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podStartSLOduration=3.820856522 podStartE2EDuration="7.531664053s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.710807531s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:04:21.530413869 +0000 UTC m=+1848.890018474" watchObservedRunningTime="2026-07-28 13:04:21.531664053 +0000 UTC m=+1848.891268677"
Jul 28 13:04:22.516443 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:22.516405    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:04:27.521627 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:27.521597    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:04:27.522186 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:27.522154    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:04:37.522722 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:37.522677    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:04:47.522109 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:47.522065    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:04:57.522120 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:04:57.522035    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:05:07.522445 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:07.522399    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:05:17.522463 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:17.522417    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:05:27.522217 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:27.522176    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:05:33.102565 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:33.102521    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 13:05:43.106652 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:43.106623    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:05:44.590774 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:44.590723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:44.607430 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:44.607395    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"]
Jul 28 13:05:44.607622 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:44.607601    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container" containerID="cri-o://201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec" gracePeriod=30
Jul 28 13:05:44.607688 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:44.607646    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kube-rbac-proxy" containerID="cri-o://683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf" gracePeriod=30
Jul 28 13:05:44.687583 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:44.687545    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:44.833318 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:44.833283    2561 generic.go:356] "Generic (PLEG): container finished" podID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerID="683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf" exitCode=2
Jul 28 13:05:44.833477 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:44.833325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" event={"ID":"5beab22d-ca46-4657-bd31-d83cc74ef79e","Type":"ContainerDied","Data":"683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf"}
Jul 28 13:05:47.289994 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:47.289932    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:47.385618 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:47.385581    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:47.517271 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:47.517220    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 28 13:05:49.454193 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.454166    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:05:49.614168 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.614131    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2wwr\" (UniqueName: \"kubernetes.io/projected/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-api-access-w2wwr\") pod \"5beab22d-ca46-4657-bd31-d83cc74ef79e\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") "
Jul 28 13:05:49.614168 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.614167    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls\") pod \"5beab22d-ca46-4657-bd31-d83cc74ef79e\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") "
Jul 28 13:05:49.614442 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.614210    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-rbac-proxy-sar-config\") pod \"5beab22d-ca46-4657-bd31-d83cc74ef79e\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") "
Jul 28 13:05:49.614442 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.614249    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-cabundle-cert\") pod \"5beab22d-ca46-4657-bd31-d83cc74ef79e\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") "
Jul 28 13:05:49.614442 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.614281    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5beab22d-ca46-4657-bd31-d83cc74ef79e-kserve-provision-location\") pod \"5beab22d-ca46-4657-bd31-d83cc74ef79e\" (UID: \"5beab22d-ca46-4657-bd31-d83cc74ef79e\") "
Jul 28 13:05:49.614676 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.614650    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5beab22d-ca46-4657-bd31-d83cc74ef79e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5beab22d-ca46-4657-bd31-d83cc74ef79e" (UID: "5beab22d-ca46-4657-bd31-d83cc74ef79e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:05:49.614676 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.614661    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5beab22d-ca46-4657-bd31-d83cc74ef79e" (UID: "5beab22d-ca46-4657-bd31-d83cc74ef79e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:05:49.614789 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.614664    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5beab22d-ca46-4657-bd31-d83cc74ef79e" (UID: "5beab22d-ca46-4657-bd31-d83cc74ef79e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:05:49.616594 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.616572    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-api-access-w2wwr" (OuterVolumeSpecName: "kube-api-access-w2wwr") pod "5beab22d-ca46-4657-bd31-d83cc74ef79e" (UID: "5beab22d-ca46-4657-bd31-d83cc74ef79e"). InnerVolumeSpecName "kube-api-access-w2wwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:05:49.616685 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.616611    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5beab22d-ca46-4657-bd31-d83cc74ef79e" (UID: "5beab22d-ca46-4657-bd31-d83cc74ef79e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:05:49.715390 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.715352    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2wwr\" (UniqueName: \"kubernetes.io/projected/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-api-access-w2wwr\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:05:49.715390 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.715385    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5beab22d-ca46-4657-bd31-d83cc74ef79e-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:05:49.715600 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.715400    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:05:49.715600 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.715412    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5beab22d-ca46-4657-bd31-d83cc74ef79e-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:05:49.715600 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.715422    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5beab22d-ca46-4657-bd31-d83cc74ef79e-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:05:49.854677 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.854632    2561 generic.go:356] "Generic (PLEG): container finished" podID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerID="201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec" exitCode=0
Jul 28 13:05:49.854860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.854697    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"
Jul 28 13:05:49.854860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.854761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" event={"ID":"5beab22d-ca46-4657-bd31-d83cc74ef79e","Type":"ContainerDied","Data":"201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec"}
Jul 28 13:05:49.854860 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.854855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6" event={"ID":"5beab22d-ca46-4657-bd31-d83cc74ef79e","Type":"ContainerDied","Data":"a32150d905864f40545115edfc2de046a001ebc279e9338e601312fa1be7a984"}
Jul 28 13:05:49.854966 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.854895    2561 scope.go:117] "RemoveContainer" containerID="683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf"
Jul 28 13:05:49.863503 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.863484    2561 scope.go:117] "RemoveContainer" containerID="201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec"
Jul 28 13:05:49.870917 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.870898    2561 scope.go:117] "RemoveContainer" containerID="952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3"
Jul 28 13:05:49.877077 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.877046    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"]
Jul 28 13:05:49.878517 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.878502    2561 scope.go:117] "RemoveContainer" containerID="683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf"
Jul 28 13:05:49.878819 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:05:49.878800    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf\": container with ID starting with 683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf not found: ID does not exist" containerID="683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf"
Jul 28 13:05:49.878875 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.878829    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf"} err="failed to get container status \"683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf\": rpc error: code = NotFound desc = could not find container \"683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf\": container with ID starting with 683547e54ac4fc002e7e1c9d078b5b3a9e18219f2d8f6baae85f985313922abf not found: ID does not exist"
Jul 28 13:05:49.878875 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.878847    2561 scope.go:117] "RemoveContainer" containerID="201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec"
Jul 28 13:05:49.879098 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:05:49.879083    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec\": container with ID starting with 201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec not found: ID does not exist" containerID="201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec"
Jul 28 13:05:49.879139 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.879102    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec"} err="failed to get container status \"201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec\": rpc error: code = NotFound desc = could not find container \"201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec\": container with ID starting with 201f4ce2b367a4cbcd08ffbf8136233dfadc9dad09558b1b87913e6b54dfe9ec not found: ID does not exist"
Jul 28 13:05:49.879139 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.879113    2561 scope.go:117] "RemoveContainer" containerID="952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3"
Jul 28 13:05:49.879343 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:05:49.879320    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3\": container with ID starting with 952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3 not found: ID does not exist" containerID="952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3"
Jul 28 13:05:49.879388 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.879351    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3"} err="failed to get container status \"952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3\": rpc error: code = NotFound desc = could not find container \"952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3\": container with ID starting with 952795316e33590527dc7275bf40e9bde3c0fcfef97df3e40c643f1e9ecaa1e3 not found: ID does not exist"
Jul 28 13:05:49.881998 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:49.881978    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-7gdd6"]
Jul 28 13:05:51.101554 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:51.101509    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" path="/var/lib/kubelet/pods/5beab22d-ca46-4657-bd31-d83cc74ef79e/volumes"
Jul 28 13:05:54.833127 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:54.833091    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:54.932202 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:05:54.932169    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:07:40.091398 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.091359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:07:40.101778 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.101725    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"]
Jul 28 13:07:40.102044 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.102032    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="storage-initializer"
Jul 28 13:07:40.102092 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.102045    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="storage-initializer"
Jul 28 13:07:40.102092 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.102062    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kube-rbac-proxy"
Jul 28 13:07:40.102092 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.102068    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kube-rbac-proxy"
Jul 28 13:07:40.102092 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.102076    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container"
Jul 28 13:07:40.102092 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.102082    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container"
Jul 28 13:07:40.102246 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.102127    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kube-rbac-proxy"
Jul 28 13:07:40.102246 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.102135    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5beab22d-ca46-4657-bd31-d83cc74ef79e" containerName="kserve-container"
Jul 28 13:07:40.105263 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.105245    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.108177 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.108141    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d919a2-kube-rbac-proxy-sar-config\""
Jul 28 13:07:40.108177 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.108172    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-d919a2-dockercfg-xvwqq\""
Jul 28 13:07:40.108401 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.108224    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:07:40.109021 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.109002    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-d919a2\""
Jul 28 13:07:40.109128 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.109037    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:07:40.109128 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.109038    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d919a2-predictor-serving-cert\""
Jul 28 13:07:40.109128 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.109041    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:07:40.116585 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.116564    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"]
Jul 28 13:07:40.191172 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.191126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e517e996-8666-47fa-9d3c-904f1c06ccb1-proxy-tls\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.191377 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.191181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.191377 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.191232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e517e996-8666-47fa-9d3c-904f1c06ccb1-kserve-provision-location\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.191377 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.191258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-cabundle-cert\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.191377 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.191332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnc25\" (UniqueName: \"kubernetes.io/projected/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-api-access-cnc25\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.291764 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.291699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnc25\" (UniqueName: \"kubernetes.io/projected/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-api-access-cnc25\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.291946 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.291808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e517e996-8666-47fa-9d3c-904f1c06ccb1-proxy-tls\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.291946 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.291837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.291946 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.291883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e517e996-8666-47fa-9d3c-904f1c06ccb1-kserve-provision-location\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.291946 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.291908    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-cabundle-cert\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.292295 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.292277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e517e996-8666-47fa-9d3c-904f1c06ccb1-kserve-provision-location\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.292546 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.292527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.292623 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.292610    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-cabundle-cert\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.294650 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.294625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e517e996-8666-47fa-9d3c-904f1c06ccb1-proxy-tls\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.302676 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.302651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnc25\" (UniqueName: \"kubernetes.io/projected/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-api-access-cnc25\") pod \"isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") " pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.416032 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.415943    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:40.546991 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:40.546966    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"]
Jul 28 13:07:40.549504 ip-10-0-138-58 kubenswrapper[2561]: W0728 13:07:40.549474    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode517e996_8666_47fa_9d3c_904f1c06ccb1.slice/crio-5a60a05f3e2865f8bab28f6ab9225ff58477119407b8f9558a929bca8de5ad3e WatchSource:0}: Error finding container 5a60a05f3e2865f8bab28f6ab9225ff58477119407b8f9558a929bca8de5ad3e: Status 404 returned error can't find the container with id 5a60a05f3e2865f8bab28f6ab9225ff58477119407b8f9558a929bca8de5ad3e
Jul 28 13:07:41.283284 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:41.283244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85" event={"ID":"e517e996-8666-47fa-9d3c-904f1c06ccb1","Type":"ContainerStarted","Data":"75fa0317894724b2128d11c992b83a2e60d2677861ecbacd551f690b44e6502f"}
Jul 28 13:07:41.283696 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:41.283339    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85" event={"ID":"e517e996-8666-47fa-9d3c-904f1c06ccb1","Type":"ContainerStarted","Data":"5a60a05f3e2865f8bab28f6ab9225ff58477119407b8f9558a929bca8de5ad3e"}
Jul 28 13:07:41.292433 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:41.292404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:07:45.299657 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:45.299627    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85_e517e996-8666-47fa-9d3c-904f1c06ccb1/storage-initializer/0.log"
Jul 28 13:07:45.300038 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:45.299666    2561 generic.go:356] "Generic (PLEG): container finished" podID="e517e996-8666-47fa-9d3c-904f1c06ccb1" containerID="75fa0317894724b2128d11c992b83a2e60d2677861ecbacd551f690b44e6502f" exitCode=1
Jul 28 13:07:45.300038 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:45.299745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85" event={"ID":"e517e996-8666-47fa-9d3c-904f1c06ccb1","Type":"ContainerDied","Data":"75fa0317894724b2128d11c992b83a2e60d2677861ecbacd551f690b44e6502f"}
Jul 28 13:07:46.305699 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:46.305671    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85_e517e996-8666-47fa-9d3c-904f1c06ccb1/storage-initializer/0.log"
Jul 28 13:07:46.306092 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:46.305791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85" event={"ID":"e517e996-8666-47fa-9d3c-904f1c06ccb1","Type":"ContainerStarted","Data":"27b60011f8a2dbde254cf69e811fefb2b3faa06c70f561e46eee2d38f4263238"}
Jul 28 13:07:49.592256 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:49.592219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:07:51.326682 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:51.326653    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85_e517e996-8666-47fa-9d3c-904f1c06ccb1/storage-initializer/1.log"
Jul 28 13:07:51.327162 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:51.327061    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85_e517e996-8666-47fa-9d3c-904f1c06ccb1/storage-initializer/0.log"
Jul 28 13:07:51.327162 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:51.327096    2561 generic.go:356] "Generic (PLEG): container finished" podID="e517e996-8666-47fa-9d3c-904f1c06ccb1" containerID="27b60011f8a2dbde254cf69e811fefb2b3faa06c70f561e46eee2d38f4263238" exitCode=1
Jul 28 13:07:51.327274 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:51.327169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85" event={"ID":"e517e996-8666-47fa-9d3c-904f1c06ccb1","Type":"ContainerDied","Data":"27b60011f8a2dbde254cf69e811fefb2b3faa06c70f561e46eee2d38f4263238"}
Jul 28 13:07:51.327274 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:51.327239    2561 scope.go:117] "RemoveContainer" containerID="75fa0317894724b2128d11c992b83a2e60d2677861ecbacd551f690b44e6502f"
Jul 28 13:07:51.327581 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:51.327556    2561 scope.go:117] "RemoveContainer" containerID="75fa0317894724b2128d11c992b83a2e60d2677861ecbacd551f690b44e6502f"
Jul 28 13:07:51.337468 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:07:51.337447    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85_kserve-ci-e2e-test(e517e996-8666-47fa-9d3c-904f1c06ccb1)\"" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85" podUID="e517e996-8666-47fa-9d3c-904f1c06ccb1"
Jul 28 13:07:51.337938 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:07:51.337907    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85_kserve-ci-e2e-test_e517e996-8666-47fa-9d3c-904f1c06ccb1_0 in pod sandbox 5a60a05f3e2865f8bab28f6ab9225ff58477119407b8f9558a929bca8de5ad3e from index: no such id: '75fa0317894724b2128d11c992b83a2e60d2677861ecbacd551f690b44e6502f'" containerID="75fa0317894724b2128d11c992b83a2e60d2677861ecbacd551f690b44e6502f"
Jul 28 13:07:51.338042 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:51.337948    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75fa0317894724b2128d11c992b83a2e60d2677861ecbacd551f690b44e6502f"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85_kserve-ci-e2e-test_e517e996-8666-47fa-9d3c-904f1c06ccb1_0 in pod sandbox 5a60a05f3e2865f8bab28f6ab9225ff58477119407b8f9558a929bca8de5ad3e from index: no such id: '75fa0317894724b2128d11c992b83a2e60d2677861ecbacd551f690b44e6502f'"
Jul 28 13:07:52.333408 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:52.333379    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85_e517e996-8666-47fa-9d3c-904f1c06ccb1/storage-initializer/1.log"
Jul 28 13:07:56.135323 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.135231    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:07:56.177886 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.177853    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"]
Jul 28 13:07:56.220005 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.219976    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:07:56.303679 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.303657    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85_e517e996-8666-47fa-9d3c-904f1c06ccb1/storage-initializer/1.log"
Jul 28 13:07:56.307351 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.307334    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:56.349429 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.349405    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85_e517e996-8666-47fa-9d3c-904f1c06ccb1/storage-initializer/1.log"
Jul 28 13:07:56.349598 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.349501    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"
Jul 28 13:07:56.349598 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.349525    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85" event={"ID":"e517e996-8666-47fa-9d3c-904f1c06ccb1","Type":"ContainerDied","Data":"5a60a05f3e2865f8bab28f6ab9225ff58477119407b8f9558a929bca8de5ad3e"}
Jul 28 13:07:56.349721 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.349596    2561 scope.go:117] "RemoveContainer" containerID="27b60011f8a2dbde254cf69e811fefb2b3faa06c70f561e46eee2d38f4263238"
Jul 28 13:07:56.420270 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.420181    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e517e996-8666-47fa-9d3c-904f1c06ccb1-proxy-tls\") pod \"e517e996-8666-47fa-9d3c-904f1c06ccb1\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") "
Jul 28 13:07:56.420270 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.420247    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e517e996-8666-47fa-9d3c-904f1c06ccb1-kserve-provision-location\") pod \"e517e996-8666-47fa-9d3c-904f1c06ccb1\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") "
Jul 28 13:07:56.420270 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.420268    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-cabundle-cert\") pod \"e517e996-8666-47fa-9d3c-904f1c06ccb1\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") "
Jul 28 13:07:56.420544 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.420374    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnc25\" (UniqueName: \"kubernetes.io/projected/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-api-access-cnc25\") pod \"e517e996-8666-47fa-9d3c-904f1c06ccb1\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") "
Jul 28 13:07:56.420544 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.420459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-rbac-proxy-sar-config\") pod \"e517e996-8666-47fa-9d3c-904f1c06ccb1\" (UID: \"e517e996-8666-47fa-9d3c-904f1c06ccb1\") "
Jul 28 13:07:56.420544 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.420511    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e517e996-8666-47fa-9d3c-904f1c06ccb1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e517e996-8666-47fa-9d3c-904f1c06ccb1" (UID: "e517e996-8666-47fa-9d3c-904f1c06ccb1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:07:56.420711 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.420607    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e517e996-8666-47fa-9d3c-904f1c06ccb1" (UID: "e517e996-8666-47fa-9d3c-904f1c06ccb1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:07:56.420711 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.420674    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e517e996-8666-47fa-9d3c-904f1c06ccb1-kserve-provision-location\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:07:56.420711 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.420688    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-cabundle-cert\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:07:56.420863 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.420795    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e517e996-8666-47fa-9d3c-904f1c06ccb1" (UID: "e517e996-8666-47fa-9d3c-904f1c06ccb1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:07:56.422472 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.422450    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e517e996-8666-47fa-9d3c-904f1c06ccb1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e517e996-8666-47fa-9d3c-904f1c06ccb1" (UID: "e517e996-8666-47fa-9d3c-904f1c06ccb1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:07:56.422587 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.422565    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-api-access-cnc25" (OuterVolumeSpecName: "kube-api-access-cnc25") pod "e517e996-8666-47fa-9d3c-904f1c06ccb1" (UID: "e517e996-8666-47fa-9d3c-904f1c06ccb1"). InnerVolumeSpecName "kube-api-access-cnc25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:07:56.522003 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.521970    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnc25\" (UniqueName: \"kubernetes.io/projected/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-api-access-cnc25\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:07:56.522003 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.521996    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e517e996-8666-47fa-9d3c-904f1c06ccb1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:07:56.522003 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.522008    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e517e996-8666-47fa-9d3c-904f1c06ccb1-proxy-tls\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:07:56.685282 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.685208    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"]
Jul 28 13:07:56.688907 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:56.688878    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d919a2-predictor-d95c8d7d6-fzm85"]
Jul 28 13:07:57.101600 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:07:57.101551    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e517e996-8666-47fa-9d3c-904f1c06ccb1" path="/var/lib/kubelet/pods/e517e996-8666-47fa-9d3c-904f1c06ccb1/volumes"
Jul 28 13:08:01.359515 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:08:01.359477    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:08:01.699266 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:08:01.699181    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:08:11.423463 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:08:11.423432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:08:16.622454 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:08:16.622402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:08:17.087258 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:08:17.087225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:08:33.142022 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:08:33.141991    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 13:08:33.143498 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:08:33.143477    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 13:09:46.816242 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:09:46.816206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:09:51.906692 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:09:51.906658    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:09:52.078625 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:09:52.078595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:09:57.050242 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:09:57.050210    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:09:57.120377 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:09:57.120350    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:12.107064 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:12.106988    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:12.178143 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:12.178105    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:18.627213 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.627176    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c5r27/must-gather-zc4rt"]
Jul 28 13:11:18.627586 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.627466    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e517e996-8666-47fa-9d3c-904f1c06ccb1" containerName="storage-initializer"
Jul 28 13:11:18.627586 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.627477    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e517e996-8666-47fa-9d3c-904f1c06ccb1" containerName="storage-initializer"
Jul 28 13:11:18.627586 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.627488    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e517e996-8666-47fa-9d3c-904f1c06ccb1" containerName="storage-initializer"
Jul 28 13:11:18.627586 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.627494    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e517e996-8666-47fa-9d3c-904f1c06ccb1" containerName="storage-initializer"
Jul 28 13:11:18.627586 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.627535    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e517e996-8666-47fa-9d3c-904f1c06ccb1" containerName="storage-initializer"
Jul 28 13:11:18.627586 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.627544    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e517e996-8666-47fa-9d3c-904f1c06ccb1" containerName="storage-initializer"
Jul 28 13:11:18.630560 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.630537    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:18.633035 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.633016    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c5r27\"/\"kube-root-ca.crt\""
Jul 28 13:11:18.633132 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.633024    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c5r27\"/\"default-dockercfg-hpw9x\""
Jul 28 13:11:18.633920 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.633890    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c5r27\"/\"openshift-service-ca.crt\""
Jul 28 13:11:18.640951 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.640932    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c5r27/must-gather-zc4rt"]
Jul 28 13:11:18.644015 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.643991    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:18.722085 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.722044    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:18.760191 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.760164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhsql\" (UniqueName: \"kubernetes.io/projected/1546c2b5-023b-4a13-a5c8-5a0090b1f542-kube-api-access-xhsql\") pod \"must-gather-zc4rt\" (UID: \"1546c2b5-023b-4a13-a5c8-5a0090b1f542\") " pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:18.760299 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.760234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1546c2b5-023b-4a13-a5c8-5a0090b1f542-must-gather-output\") pod \"must-gather-zc4rt\" (UID: \"1546c2b5-023b-4a13-a5c8-5a0090b1f542\") " pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:18.860885 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.860849    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1546c2b5-023b-4a13-a5c8-5a0090b1f542-must-gather-output\") pod \"must-gather-zc4rt\" (UID: \"1546c2b5-023b-4a13-a5c8-5a0090b1f542\") " pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:18.861049 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.860912    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhsql\" (UniqueName: \"kubernetes.io/projected/1546c2b5-023b-4a13-a5c8-5a0090b1f542-kube-api-access-xhsql\") pod \"must-gather-zc4rt\" (UID: \"1546c2b5-023b-4a13-a5c8-5a0090b1f542\") " pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:18.861193 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.861171    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1546c2b5-023b-4a13-a5c8-5a0090b1f542-must-gather-output\") pod \"must-gather-zc4rt\" (UID: \"1546c2b5-023b-4a13-a5c8-5a0090b1f542\") " pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:18.871658 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.871629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhsql\" (UniqueName: \"kubernetes.io/projected/1546c2b5-023b-4a13-a5c8-5a0090b1f542-kube-api-access-xhsql\") pod \"must-gather-zc4rt\" (UID: \"1546c2b5-023b-4a13-a5c8-5a0090b1f542\") " pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:18.939932 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:18.939851    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:19.067487 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:19.067462    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c5r27/must-gather-zc4rt"]
Jul 28 13:11:19.069322 ip-10-0-138-58 kubenswrapper[2561]: W0728 13:11:19.069296    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1546c2b5_023b_4a13_a5c8_5a0090b1f542.slice/crio-65bae08929a88faa38215c73064029d0c2bbdb0072f16a8a7dc05509622ba26d WatchSource:0}: Error finding container 65bae08929a88faa38215c73064029d0c2bbdb0072f16a8a7dc05509622ba26d: Status 404 returned error can't find the container with id 65bae08929a88faa38215c73064029d0c2bbdb0072f16a8a7dc05509622ba26d
Jul 28 13:11:19.071342 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:19.071325    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:11:19.109123 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:19.109094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c5r27/must-gather-zc4rt" event={"ID":"1546c2b5-023b-4a13-a5c8-5a0090b1f542","Type":"ContainerStarted","Data":"65bae08929a88faa38215c73064029d0c2bbdb0072f16a8a7dc05509622ba26d"}
Jul 28 13:11:26.146800 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:26.146762    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c5r27/must-gather-zc4rt" event={"ID":"1546c2b5-023b-4a13-a5c8-5a0090b1f542","Type":"ContainerStarted","Data":"994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed"}
Jul 28 13:11:26.146800 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:26.146808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c5r27/must-gather-zc4rt" event={"ID":"1546c2b5-023b-4a13-a5c8-5a0090b1f542","Type":"ContainerStarted","Data":"2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba"}
Jul 28 13:11:26.167414 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:26.167363    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c5r27/must-gather-zc4rt" podStartSLOduration=1.9186317320000001 podStartE2EDuration="8.167347849s" totalImagesPullingTime="6.248716117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:11:26.165757165 +0000 UTC m=+2273.525361764" watchObservedRunningTime="2026-07-28 13:11:26.167347849 +0000 UTC m=+2273.526952459"
Jul 28 13:11:48.236242 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:48.236151    2561 generic.go:356] "Generic (PLEG): container finished" podID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" containerID="2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba" exitCode=0
Jul 28 13:11:48.236242 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:48.236230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c5r27/must-gather-zc4rt" event={"ID":"1546c2b5-023b-4a13-a5c8-5a0090b1f542","Type":"ContainerDied","Data":"2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba"}
Jul 28 13:11:48.236665 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:48.236572    2561 scope.go:117] "RemoveContainer" containerID="2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba"
Jul 28 13:11:48.869313 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:48.869288    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c5r27_must-gather-zc4rt_1546c2b5-023b-4a13-a5c8-5a0090b1f542/gather/0.log"
Jul 28 13:11:49.334411 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:49.334349    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:49.505158 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:49.505117    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:49.605874 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:49.605770    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:52.173473 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:52.173444    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h72fp_14ee1edf-eb79-49a6-b644-7fb9f2bf93d0/global-pull-secret-syncer/0.log"
Jul 28 13:11:52.349996 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:52.349968    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log"
Jul 28 13:11:52.495153 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:52.495076    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-58.ec2.internal_3d8f03f8e6759f6f8b7bccae6a6de85c/haproxy/0.log"
Jul 28 13:11:54.311463 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.311423    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-c5r27/must-gather-zc4rt"]
Jul 28 13:11:54.311912 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.311622    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-c5r27/must-gather-zc4rt" podUID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" containerName="copy" containerID="cri-o://994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed" gracePeriod=2
Jul 28 13:11:54.318236 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.318209    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-c5r27/must-gather-zc4rt"]
Jul 28 13:11:54.552038 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.552016    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c5r27_must-gather-zc4rt_1546c2b5-023b-4a13-a5c8-5a0090b1f542/copy/0.log"
Jul 28 13:11:54.554520 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.554501    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:54.556787 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:11:54.556762    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zc4rt\" is forbidden: User \"system:node:ip-10-0-138-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c5r27\": no relationship found between node 'ip-10-0-138-58.ec2.internal' and this object" podUID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:54.669660 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.669558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1546c2b5-023b-4a13-a5c8-5a0090b1f542-must-gather-output\") pod \"1546c2b5-023b-4a13-a5c8-5a0090b1f542\" (UID: \"1546c2b5-023b-4a13-a5c8-5a0090b1f542\") "
Jul 28 13:11:54.669660 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.669652    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhsql\" (UniqueName: \"kubernetes.io/projected/1546c2b5-023b-4a13-a5c8-5a0090b1f542-kube-api-access-xhsql\") pod \"1546c2b5-023b-4a13-a5c8-5a0090b1f542\" (UID: \"1546c2b5-023b-4a13-a5c8-5a0090b1f542\") "
Jul 28 13:11:54.671075 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.671051    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1546c2b5-023b-4a13-a5c8-5a0090b1f542-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "1546c2b5-023b-4a13-a5c8-5a0090b1f542" (UID: "1546c2b5-023b-4a13-a5c8-5a0090b1f542"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:11:54.671992 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.671969    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1546c2b5-023b-4a13-a5c8-5a0090b1f542-kube-api-access-xhsql" (OuterVolumeSpecName: "kube-api-access-xhsql") pod "1546c2b5-023b-4a13-a5c8-5a0090b1f542" (UID: "1546c2b5-023b-4a13-a5c8-5a0090b1f542"). InnerVolumeSpecName "kube-api-access-xhsql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:11:54.770241 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.770210    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhsql\" (UniqueName: \"kubernetes.io/projected/1546c2b5-023b-4a13-a5c8-5a0090b1f542-kube-api-access-xhsql\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:11:54.770241 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:54.770233    2561 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1546c2b5-023b-4a13-a5c8-5a0090b1f542-must-gather-output\") on node \"ip-10-0-138-58.ec2.internal\" DevicePath \"\""
Jul 28 13:11:55.042548 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.042517    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:55.101794 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.101769    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" path="/var/lib/kubelet/pods/1546c2b5-023b-4a13-a5c8-5a0090b1f542/volumes"
Jul 28 13:11:55.143253 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.143230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kt6wr_bc2806da-0277-4109-a56b-47fddeef6f1d/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:55.262790 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.262765    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c5r27_must-gather-zc4rt_1546c2b5-023b-4a13-a5c8-5a0090b1f542/copy/0.log"
Jul 28 13:11:55.263137 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.263113    2561 generic.go:356] "Generic (PLEG): container finished" podID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" containerID="994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed" exitCode=143
Jul 28 13:11:55.263205 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.263163    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c5r27/must-gather-zc4rt"
Jul 28 13:11:55.263205 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.263186    2561 scope.go:117] "RemoveContainer" containerID="994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed"
Jul 28 13:11:55.270995 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.270886    2561 scope.go:117] "RemoveContainer" containerID="2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba"
Jul 28 13:11:55.283353 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.283330    2561 scope.go:117] "RemoveContainer" containerID="994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed"
Jul 28 13:11:55.283627 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:11:55.283608    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed\": container with ID starting with 994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed not found: ID does not exist" containerID="994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed"
Jul 28 13:11:55.283674 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.283636    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed"} err="failed to get container status \"994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed\": rpc error: code = NotFound desc = could not find container \"994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed\": container with ID starting with 994cb152c3ef4114399cd9870e44635da78ac1ec71ad2d7031e326b33002c6ed not found: ID does not exist"
Jul 28 13:11:55.283674 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.283653    2561 scope.go:117] "RemoveContainer" containerID="2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba"
Jul 28 13:11:55.283925 ip-10-0-138-58 kubenswrapper[2561]: E0728 13:11:55.283907    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba\": container with ID starting with 2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba not found: ID does not exist" containerID="2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba"
Jul 28 13:11:55.283976 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:55.283930    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba"} err="failed to get container status \"2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba\": rpc error: code = NotFound desc = could not find container \"2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba\": container with ID starting with 2054989f16c4016c3538663d2ee3f927c5de6de35c9dab4f0dd5a484438d4eba not found: ID does not exist"
Jul 28 13:11:56.959516 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:56.959472    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jkbbm_a69c6607-ffb6-4416-9e86-9ab57896b3aa/node-exporter/0.log"
Jul 28 13:11:57.001269 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:57.001245    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jkbbm_a69c6607-ffb6-4416-9e86-9ab57896b3aa/kube-rbac-proxy/0.log"
Jul 28 13:11:57.033199 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:57.033174    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jkbbm_a69c6607-ffb6-4416-9e86-9ab57896b3aa/init-textfile/0.log"
Jul 28 13:11:57.171624 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:57.171539    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6a79ea7e-5093-47be-b591-ee72343c0c94/prometheus/0.log"
Jul 28 13:11:57.201030 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:57.201009    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6a79ea7e-5093-47be-b591-ee72343c0c94/config-reloader/0.log"
Jul 28 13:11:57.231858 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:57.231839    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6a79ea7e-5093-47be-b591-ee72343c0c94/thanos-sidecar/0.log"
Jul 28 13:11:57.283370 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:57.283340    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6a79ea7e-5093-47be-b591-ee72343c0c94/kube-rbac-proxy-web/0.log"
Jul 28 13:11:57.330375 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:57.330352    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6a79ea7e-5093-47be-b591-ee72343c0c94/kube-rbac-proxy/0.log"
Jul 28 13:11:57.357850 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:57.357827    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6a79ea7e-5093-47be-b591-ee72343c0c94/kube-rbac-proxy-thanos/0.log"
Jul 28 13:11:57.384851 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:57.384826    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6a79ea7e-5093-47be-b591-ee72343c0c94/init-config-reloader/0.log"
Jul 28 13:11:59.499756 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.499709    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/1.log"
Jul 28 13:11:59.503179 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.503161    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-jhd5j_39287a4b-97a2-495a-a4aa-e284ef517bf6/console-operator/2.log"
Jul 28 13:11:59.753487 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.753407    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"]
Jul 28 13:11:59.753683 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.753669    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" containerName="gather"
Jul 28 13:11:59.753756 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.753685    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" containerName="gather"
Jul 28 13:11:59.753756 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.753696    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" containerName="copy"
Jul 28 13:11:59.753756 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.753702    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" containerName="copy"
Jul 28 13:11:59.753858 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.753788    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" containerName="gather"
Jul 28 13:11:59.753858 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.753798    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1546c2b5-023b-4a13-a5c8-5a0090b1f542" containerName="copy"
Jul 28 13:11:59.758596 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.758578    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:11:59.761252 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.761231    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4r45x\"/\"default-dockercfg-8kmjh\""
Jul 28 13:11:59.761696 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.761679    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4r45x\"/\"openshift-service-ca.crt\""
Jul 28 13:11:59.762245 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.762230    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4r45x\"/\"kube-root-ca.crt\""
Jul 28 13:11:59.767428 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.767406    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"]
Jul 28 13:11:59.912304 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.912275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-sys\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:11:59.912455 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.912314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-proc\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:11:59.912455 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.912331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-lib-modules\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:11:59.912455 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.912398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-podres\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:11:59.912455 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.912439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9kp\" (UniqueName: \"kubernetes.io/projected/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-kube-api-access-5b9kp\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:11:59.973206 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:11:59.973171    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-w58tj_14b46c60-e496-4c48-baa3-1780844ca48d/download-server/0.log"
Jul 28 13:12:00.012905 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.012815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-sys\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.012905 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.012865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-proc\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.012905 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.012884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-lib-modules\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.013160 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.012921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-podres\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.013160 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.012950    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-sys\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.013160 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.012966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-proc\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.013160 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.012952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9kp\" (UniqueName: \"kubernetes.io/projected/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-kube-api-access-5b9kp\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.013160 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.013064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-podres\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.013160 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.013082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-lib-modules\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.022365 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.022339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9kp\" (UniqueName: \"kubernetes.io/projected/cb7acf95-ecbd-49aa-9cd4-d2097796f99c-kube-api-access-5b9kp\") pod \"perf-node-gather-daemonset-jjzdf\" (UID: \"cb7acf95-ecbd-49aa-9cd4-d2097796f99c\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.068480 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.068446    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.202495 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.202454    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"]
Jul 28 13:12:00.206539 ip-10-0-138-58 kubenswrapper[2561]: W0728 13:12:00.206517    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb7acf95_ecbd_49aa_9cd4_d2097796f99c.slice/crio-c9873a22d9c9e34433ef1d3352ad86a048901f17c50980cc90501e8cd757627d WatchSource:0}: Error finding container c9873a22d9c9e34433ef1d3352ad86a048901f17c50980cc90501e8cd757627d: Status 404 returned error can't find the container with id c9873a22d9c9e34433ef1d3352ad86a048901f17c50980cc90501e8cd757627d
Jul 28 13:12:00.289489 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.289457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf" event={"ID":"cb7acf95-ecbd-49aa-9cd4-d2097796f99c","Type":"ContainerStarted","Data":"08c1399f141042e1e997a7418ddf5852ed195e049d5d1505ff55f0a8b75acaab"}
Jul 28 13:12:00.289838 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.289506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf" event={"ID":"cb7acf95-ecbd-49aa-9cd4-d2097796f99c","Type":"ContainerStarted","Data":"c9873a22d9c9e34433ef1d3352ad86a048901f17c50980cc90501e8cd757627d"}
Jul 28 13:12:00.289838 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.289586    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:00.490786 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:00.490754    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-bvwkp_629f00bc-22c6-439d-9439-6113ad8ccef4/volume-data-source-validator/0.log"
Jul 28 13:12:01.382538 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:01.382513    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dmbdk_84a04ae8-3976-4608-92b8-26002d12fb16/dns/0.log"
Jul 28 13:12:01.407749 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:01.407705    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dmbdk_84a04ae8-3976-4608-92b8-26002d12fb16/kube-rbac-proxy/0.log"
Jul 28 13:12:01.569918 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:01.569891    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gwgtq_f11de13b-b398-4dc5-a41d-7e08de507c51/dns-node-resolver/0.log"
Jul 28 13:12:02.081472 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:02.081440    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-28qkt_fc255e86-f3aa-4374-9f7a-c4de06be12a6/node-ca/0.log"
Jul 28 13:12:03.259784 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:03.259756    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6zc4d_b6494845-81de-42c7-b8de-2c5419e0ba31/serve-healthcheck-canary/0.log"
Jul 28 13:12:03.705014 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:03.704987    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b7g7l_54ff797e-a55b-4020-8e90-97d5bb21db66/kube-rbac-proxy/0.log"
Jul 28 13:12:03.729609 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:03.729586    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b7g7l_54ff797e-a55b-4020-8e90-97d5bb21db66/exporter/0.log"
Jul 28 13:12:03.754960 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:03.754941    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b7g7l_54ff797e-a55b-4020-8e90-97d5bb21db66/extractor/0.log"
Jul 28 13:12:06.303085 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:06.303057    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf"
Jul 28 13:12:06.327904 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:06.327845    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-jjzdf" podStartSLOduration=7.32782667 podStartE2EDuration="7.32782667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:12:00.306375837 +0000 UTC m=+2307.665980443" watchObservedRunningTime="2026-07-28 13:12:06.32782667 +0000 UTC m=+2313.687431278"
Jul 28 13:12:06.469528 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:06.469502    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2kv82_b452d5ec-0282-4dd5-9361-0104fce274e1/s3-init/0.log"
Jul 28 13:12:06.488393 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:06.488368    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2kv82_b452d5ec-0282-4dd5-9361-0104fce274e1/download-hf-model/0.log"
Jul 28 13:12:11.110429 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:11.110402    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf_56dbca26-1e1f-47ec-aa9c-88881434fa9e/kube-storage-version-migrator-operator/1.log"
Jul 28 13:12:11.111215 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:11.111196    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wc4hf_56dbca26-1e1f-47ec-aa9c-88881434fa9e/kube-storage-version-migrator-operator/0.log"
Jul 28 13:12:12.102207 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:12.102135    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5d5j_20f422dd-5142-4477-a522-4b50a217e8b4/kube-multus-additional-cni-plugins/0.log"
Jul 28 13:12:12.134028 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:12.134003    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5d5j_20f422dd-5142-4477-a522-4b50a217e8b4/egress-router-binary-copy/0.log"
Jul 28 13:12:12.171580 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:12.171557    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5d5j_20f422dd-5142-4477-a522-4b50a217e8b4/cni-plugins/0.log"
Jul 28 13:12:12.198346 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:12.198325    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5d5j_20f422dd-5142-4477-a522-4b50a217e8b4/bond-cni-plugin/0.log"
Jul 28 13:12:12.224957 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:12.224927    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5d5j_20f422dd-5142-4477-a522-4b50a217e8b4/routeoverride-cni/0.log"
Jul 28 13:12:12.249189 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:12.249166    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5d5j_20f422dd-5142-4477-a522-4b50a217e8b4/whereabouts-cni-bincopy/0.log"
Jul 28 13:12:12.275220 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:12.275201    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5d5j_20f422dd-5142-4477-a522-4b50a217e8b4/whereabouts-cni/0.log"
Jul 28 13:12:12.804219 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:12.804193    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-m57qs_5635b25c-2a48-4626-aeed-a2f7589b5074/kube-multus/0.log"
Jul 28 13:12:12.977017 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:12.976986    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tjcgm_18584de2-6d71-420c-a072-9b4ff6950071/network-metrics-daemon/0.log"
Jul 28 13:12:13.007384 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:13.007353    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tjcgm_18584de2-6d71-420c-a072-9b4ff6950071/kube-rbac-proxy/0.log"
Jul 28 13:12:14.286563 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:14.286538    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbd9x_f3e83f2c-efb6-4e20-a23c-5a24d97cb18e/ovn-controller/0.log"
Jul 28 13:12:14.306876 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:14.306855    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbd9x_f3e83f2c-efb6-4e20-a23c-5a24d97cb18e/ovn-acl-logging/0.log"
Jul 28 13:12:14.331917 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:14.331893    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbd9x_f3e83f2c-efb6-4e20-a23c-5a24d97cb18e/kube-rbac-proxy-node/0.log"
Jul 28 13:12:14.360161 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:14.360139    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbd9x_f3e83f2c-efb6-4e20-a23c-5a24d97cb18e/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 13:12:14.382046 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:14.382024    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbd9x_f3e83f2c-efb6-4e20-a23c-5a24d97cb18e/northd/0.log"
Jul 28 13:12:14.404293 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:14.404275    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbd9x_f3e83f2c-efb6-4e20-a23c-5a24d97cb18e/nbdb/0.log"
Jul 28 13:12:14.429127 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:14.429106    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbd9x_f3e83f2c-efb6-4e20-a23c-5a24d97cb18e/sbdb/0.log"
Jul 28 13:12:14.477770 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:14.477726    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qbd9x_f3e83f2c-efb6-4e20-a23c-5a24d97cb18e/ovnkube-controller/0.log"
Jul 28 13:12:15.491691 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:15.491663    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-nqdtk_c231cdd2-c1fd-42fe-a6ed-1cde78591cc1/check-endpoints/0.log"
Jul 28 13:12:15.567007 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:15.566983    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zn6gv_04e859b2-becc-46c1-a587-64e662d550dd/network-check-target-container/0.log"
Jul 28 13:12:16.577968 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:16.577939    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xp9lg_2c4c911a-2c4b-4230-9e13-13f2810272c0/iptables-alerter/0.log"
Jul 28 13:12:17.309245 ip-10-0-138-58 kubenswrapper[2561]: I0728 13:12:17.309220    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-zl6cw_39f606ae-f5c0-4d7b-bd8e-f927a72245ea/tuned/0.log"