Jul 28 12:31:46.819913 ip-10-0-142-139 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 12:31:46.819925 ip-10-0-142-139 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 12:31:46.819935 ip-10-0-142-139 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 12:31:46.820250 ip-10-0-142-139 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 12:31:57.068913 ip-10-0-142-139 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 12:31:57.068929 ip-10-0-142-139 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4cad448a1de94a3ea5223873d96bcac0 --
Jul 28 12:33:50.458839 ip-10-0-142-139 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 12:33:50.778209 ip-10-0-142-139 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:33:50.778209 ip-10-0-142-139 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 12:33:50.778209 ip-10-0-142-139 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:33:50.778209 ip-10-0-142-139 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 12:33:50.778209 ip-10-0-142-139 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:33:50.778886 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.778265    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 12:33:50.784884 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784863    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:50.784884 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784880    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:50.784884 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784884    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:50.784884 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784887    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:50.784884 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784891    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784895    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784898    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784901    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784904    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784907    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784910    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784913    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784915    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784918    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784920    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784923    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784926    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784928    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784931    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784933    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784936    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784939    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784941    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784944    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:50.785073 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784947    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784950    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784952    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784955    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784958    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784962    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784965    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784968    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784971    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784973    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784978    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784981    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784984    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784987    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784990    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784993    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784996    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.784999    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785001    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:50.785560 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785004    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785007    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785010    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785013    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785015    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785018    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785020    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785023    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785026    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785029    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785031    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785034    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785037    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785039    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785041    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785044    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785047    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785049    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785051    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:50.786068 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785054    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785057    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785059    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785062    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785065    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785068    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785071    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785074    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785078    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785081    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785084    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785087    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785090    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785092    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785095    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785098    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785100    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785103    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785105    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785108    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:50.786526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785110    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785119    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785122    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785124    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785127    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785129    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785132    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785134    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785137    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785139    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785142    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785145    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785147    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785150    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785153    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785157    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785160    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785890    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785897    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:50.787025 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785900    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785904    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785907    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785910    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785913    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785916    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785919    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785921    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785924    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785926    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785929    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785931    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785934    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785937    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785939    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785943    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785945    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785948    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785951    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785953    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:50.787488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785956    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785958    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785961    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785963    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785966    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785968    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785970    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785973    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785976    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785979    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785981    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785984    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785986    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785989    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785992    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785994    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785996    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.785999    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786001    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786004    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:50.787999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786007    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786009    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786012    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786014    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786017    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786019    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786021    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786024    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786033    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786035    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786039    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786042    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786045    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786048    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786051    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786053    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786056    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786059    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786062    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786064    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:50.788494 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786067    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786071    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786074    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786078    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786081    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786085    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786087    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786090    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786093    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786096    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786099    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786102    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786105    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786107    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786110    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786113    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786116    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786118    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786121    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:50.788999 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786123    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786126    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786129    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786131    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786134    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786136    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786139    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786143    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786146    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786150    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786154    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786157    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786159    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786162    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786165    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786168    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786170    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786173    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786595    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:50.789464 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786602    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786605    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786608    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786611    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786614    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786617    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786626    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786629    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786634    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786637    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786640    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786643    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786646    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786649    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786653    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786657    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786659    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786662    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786665    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:50.789945 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786667    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786670    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786673    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786675    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786678    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786681    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786683    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786685    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786688    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786691    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786693    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786696    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786698    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786701    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786703    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786706    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786709    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786711    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786714    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786716    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:50.790411 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786719    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786721    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786723    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786726    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786728    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786732    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786735    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786738    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786741    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786744    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786746    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786749    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786751    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786754    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786756    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786759    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786762    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786765    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786768    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:50.791078 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786772    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786774    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786777    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786780    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786782    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786786    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786788    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786791    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786794    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786796    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786798    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786801    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786803    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786806    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786808    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786811    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786815    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786817    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786819    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786822    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:50.791564 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786825    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786827    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786830    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786833    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786836    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786838    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786841    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786844    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786846    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786849    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786851    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786854    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786856    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786859    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786861    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786863    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786866    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786868    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786870    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.786873    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786936    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 12:33:50.792059 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786943    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786948    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786953    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786958    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786963    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786970    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786976    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786980    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786984    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786987    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786990    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786993    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786996    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.786999    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787002    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787010    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787013    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787018    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787021    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787024    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787027    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787031    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787035    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 12:33:50.792576 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787038    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787042    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787045    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787048    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787051    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787054    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787057    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787060    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787064    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787067    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787070    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787073    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787076    2569 flags.go:64] FLAG: --enable-server="true"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787079    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787083    2569 flags.go:64] FLAG: --event-burst="100"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787086    2569 flags.go:64] FLAG: --event-qps="50"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787089    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787092    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787095    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787099    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787103    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787107    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787109    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787112    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787116    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 12:33:50.793131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787119    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787123    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787126    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787128    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787131    2569 flags.go:64] FLAG: --feature-gates=""
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787135    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787138    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787141    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787144    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787147    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787150    2569 flags.go:64] FLAG: --help="false"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787152    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-142-139.ec2.internal"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787156    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787159    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787162    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787165    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787168    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787171    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787174    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787177    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787179    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787182    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787185    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787188    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 28 12:33:50.793797 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787191    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787194    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787197    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787201    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787204    2569 flags.go:64] FLAG: --lock-file=""
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787207    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787210    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787213    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787218    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787221    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787224    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787227    2569 flags.go:64] FLAG: --logging-format="text"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787230    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787233    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787235    2569 flags.go:64] FLAG: --manifest-url=""
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787238    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787242    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787245    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787249    2569 flags.go:64] FLAG: --max-pods="110"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787251    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787255    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787258    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787261    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787264    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787266    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 12:33:50.794386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787269    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787276    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787279    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787282    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787285    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787287    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787294    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787298    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787302    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787305    2569 flags.go:64] FLAG: --port="10250"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787310    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787312    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a5626294b98d8b21"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787316    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787319    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787323    2569 flags.go:64] FLAG: --register-node="true"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787326    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787329    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787333    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787335    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787338    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787342    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787345    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787348    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787351    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787353    2569 flags.go:64] FLAG: --runonce="false"
Jul 28 12:33:50.795061 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787356    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787359    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787362    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787365    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787368    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787370    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787373    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787376    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787379    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787382    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787385    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787388    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787391    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787394    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787396    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787401    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787404    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787408    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787412    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787415    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787418    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787422    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787425    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787428    2569 flags.go:64] FLAG: --v="2"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787437    2569 flags.go:64] FLAG: --version="false"
Jul 28 12:33:50.795736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787442    2569 flags.go:64] FLAG: --vmodule=""
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787445    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.787449    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787592    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787597    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787600    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787603    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787606    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787609    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787611    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787614    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787616    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787620    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787624    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787626    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787629    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787632    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787634    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787637    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:50.796357 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787640    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787642    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787644    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787647    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787651    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787655    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787657    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787660    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787663    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787668    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787670    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787673    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787675    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787678    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787682    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787684    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787687    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787689    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787692    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787694    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:50.796839 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787697    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787699    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787702    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787705    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787707    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787710    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787713    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787715    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787718    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787720    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787723    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787726    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787731    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787734    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787738    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787741    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787744    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787749    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787752    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787754    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:50.797309 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787757    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787762    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787765    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787767    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787770    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787772    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787775    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787778    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787781    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787783    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787785    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787788    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787791    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787793    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787796    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787798    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787801    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787803    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787806    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787809    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:50.797799 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787811    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787814    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787816    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787818    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787821    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787823    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787826    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787828    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787831    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787835    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787837    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787840    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787843    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787846    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787849    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787851    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787854    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787857    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787859    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787863    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:50.798283 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787867    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787870    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.787873    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.788395    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.794865    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.794881    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.795572    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.795590    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795655    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795660    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795664    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795667    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795670    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795673    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795676    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:50.798809 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795679    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795682    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795685    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795688    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795691    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795694    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795697    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795700    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795702    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795706    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795710    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795713    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795716    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795718    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795721    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795724    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795727    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795730    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795734    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:50.799186 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795737    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795741    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795744    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795747    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795751    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795754    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795758    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795761    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795763    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795766    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795769    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795771    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795774    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795777    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795780    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795782    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795785    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795787    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795790    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795793    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:50.799774 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795796    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795798    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795801    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795803    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795806    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795808    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795811    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795813    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795816    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795818    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795821    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795823    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795826    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795829    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795831    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795834    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795837    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795840    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795843    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795846    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:50.800306 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795848    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795852    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795855    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795858    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795861    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795863    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795866    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795869    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795871    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795874    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795876    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795879    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795881    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795884    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795886    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795889    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795891    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795893    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795896    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795898    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:50.800953 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795901    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795903    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795906    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795908    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795911    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795913    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795916    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795919    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795921    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795925    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795928    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795930    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.795933    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.795938    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796047    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:33:50.801696 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796051    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796054    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796057    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796060    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796063    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796065    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796068    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796071    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796073    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796076    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796078    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796081    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796084    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796086    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796089    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796091    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796094    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796097    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796099    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796102    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:33:50.802280 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796104    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796107    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796109    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796112    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796114    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796117    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796120    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796123    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796125    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796127    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796130    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796132    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796136    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796139    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796143    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796146    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796148    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796152    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796155    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:33:50.802896 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796158    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796161    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796164    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796167    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796169    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796172    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796175    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796177    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796180    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796182    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796184    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796187    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796190    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796192    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796195    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796197    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796200    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796203    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796205    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796208    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:33:50.803646 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796210    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796213    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796216    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796218    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796221    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796223    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796227    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796231    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796233    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796236    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796238    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796241    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796243    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796245    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796248    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796251    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796253    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796256    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796258    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:33:50.804425 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796261    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796263    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796266    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796268    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796271    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796273    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796276    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796278    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796281    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796284    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796287    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796289    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796292    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796294    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796297    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796301    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796303    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796306    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796309    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:33:50.804956 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:50.796311    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:33:50.805430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.796317    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:33:50.805430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.796415    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 12:33:50.805430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.798383    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 12:33:50.805430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.799451    2569 server.go:1019] "Starting client certificate rotation"
Jul 28 12:33:50.805430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.799582    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:33:50.805430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.799635    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:33:50.805430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.801630    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:33:50.807316 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.807294    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:33:50.814873 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.814858    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 28 12:33:50.817859 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.817842    2569 log.go:25] "Validated CRI v1 image API"
Jul 28 12:33:50.818689 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.818672    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 12:33:50.821084 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.821062    2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a189fef4-7367-4d14-bf53-4db94614e48c:/dev/nvme0n1p4 fbd41743-65a8-4793-a207-24893056767c:/dev/nvme0n1p3]
Jul 28 12:33:50.821157 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.821082    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 12:33:50.826137 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.826017    2569 manager.go:216] Machine: {Timestamp:2026-07-28 12:33:50.824686299 +0000 UTC m=+0.266651509 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097892 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec254033b3c625712882fb2c25df7e06 SystemUUID:ec254033-b3c6-2571-2882-fb2c25df7e06 BootID:4cad448a-1de9-4a3e-a522-3873d96bcac0 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4b:5a:8a:6d:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4b:5a:8a:6d:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:26:b2:5e:99:b4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 12:33:50.826137 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.826130    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 12:33:50.826286 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.826238    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 12:33:50.829023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.828995    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 12:33:50.829190 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.829025    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 12:33:50.829635 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.829623    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 12:33:50.829690 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.829639    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 12:33:50.829690 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.829650    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:33:50.830240 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.830229    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:33:50.830968 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.830956    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 12:33:50.831390 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.831372    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:33:50.832631 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.832618    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:33:50.832747 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.832736    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 12:33:50.832799 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.832773    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 12:33:50.832799 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.832788    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 12:33:50.832897 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.832809    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 12:33:50.832897 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.832822    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 28 12:33:50.832897 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.832834    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 12:33:50.833193 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.833178    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:33:50.833257 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.833204    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:33:50.835072 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.835046    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 12:33:50.836864 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.836730    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 12:33:50.836864 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.836780    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 12:33:50.838176 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838158    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 12:33:50.838176 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838179    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 12:33:50.838273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838188    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 12:33:50.838273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838198    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 12:33:50.838273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838209    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 12:33:50.838273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838214    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 12:33:50.838273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838223    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 12:33:50.838273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838227    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 12:33:50.838273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838234    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 12:33:50.838273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838239    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 12:33:50.838273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838255    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 12:33:50.838273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838260    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 12:33:50.838876 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838862    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 12:33:50.838876 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.838873    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 12:33:50.841968 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.841954    2569 server.go:1268] "Started kubelet"
Jul 28 12:33:50.842064 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.842030    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 12:33:50.842154 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.842103    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 12:33:50.842221 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.842179    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 12:33:50.842806 ip-10-0-142-139 systemd[1]: Started Kubernetes Kubelet.
Jul 28 12:33:50.843812 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.843790    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 12:33:50.844166 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.844137    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 12:33:50.844553 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.844469    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 12:33:50.844553 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.844478    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 12:33:50.844696 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.844642    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 12:33:50.849020 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.848198    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-139.ec2.internal.18c6742b1e6e07c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-139.ec2.internal,UID:ip-10-0-142-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-139.ec2.internal,},FirstTimestamp:2026-07-28 12:33:50.841923529 +0000 UTC m=+0.283888743,LastTimestamp:2026-07-28 12:33:50.841923529 +0000 UTC m=+0.283888743,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-139.ec2.internal,}"
Jul 28 12:33:50.849916 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.849897    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 12:33:50.850011 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.849995    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 12:33:50.850087 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.850011    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 12:33:50.850087 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.850080    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:50.850636 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.850608    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 12:33:50.850734 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.850618    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 12:33:50.850814 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.850736    2569 factory.go:55] Registering systemd factory
Jul 28 12:33:50.850814 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.850750    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 28 12:33:50.850814 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.850736    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 12:33:50.850952 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.850905    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 12:33:50.850952 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.850913    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 12:33:50.851031 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.850961    2569 factory.go:153] Registering CRI-O factory
Jul 28 12:33:50.851031 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.850971    2569 factory.go:223] Registration of the crio container factory successfully
Jul 28 12:33:50.851031 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.851029    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 12:33:50.851176 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.851053    2569 factory.go:103] Registering Raw factory
Jul 28 12:33:50.851176 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.851068    2569 manager.go:1196] Started watching for new ooms in manager
Jul 28 12:33:50.851393 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.851379    2569 manager.go:349] Starting recovery of all containers
Jul 28 12:33:50.854727 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.854700    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 12:33:50.854823 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.854774    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 12:33:50.856033 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.856015    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-brp9n"
Jul 28 12:33:50.860097 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.860063    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 12:33:50.862926 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.862907    2569 manager.go:354] Recovery completed
Jul 28 12:33:50.863888 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.863867    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-brp9n"
Jul 28 12:33:50.864624 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.864598    2569 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 12:33:50.867602 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.867588    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:50.871012 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.870993    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:50.871082 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.871025    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:50.871082 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.871039    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:50.871591 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.871574    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 12:33:50.871591 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.871584    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:50.871591 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.871592    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 12:33:50.871781 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.871608    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:33:50.872224 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.872207    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:50.872277 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.872238    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:50.872277 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.872252    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:50.872990 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.872978    2569 policy_none.go:49] "None policy: Start"
Jul 28 12:33:50.873047 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.872995    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 12:33:50.873047 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.873010    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 12:33:50.873275 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.873214    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-139.ec2.internal.18c6742b2029e0b6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-139.ec2.internal,UID:ip-10-0-142-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-139.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-139.ec2.internal,},FirstTimestamp:2026-07-28 12:33:50.87101151 +0000 UTC m=+0.312976725,LastTimestamp:2026-07-28 12:33:50.87101151 +0000 UTC m=+0.312976725,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-139.ec2.internal,}"
Jul 28 12:33:50.874062 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.874049    2569 policy_none.go:47] "Start"
Jul 28 12:33:50.909212 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.909084    2569 manager.go:337] "Starting Device Plugin manager"
Jul 28 12:33:50.909310 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.909242    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 12:33:50.909310 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.909253    2569 server.go:87] "Starting device plugin registration server"
Jul 28 12:33:50.909460 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.909449    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 12:33:50.909506 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.909461    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 12:33:50.909768 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.909603    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 12:33:50.909768 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.909681    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 12:33:50.909768 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:50.909695    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 12:33:50.910756 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.910738    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 12:33:50.910830 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:50.910789    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:51.009859 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.009829    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:51.010891 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.010870    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:51.010980 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.010905    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:51.010980 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.010919    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:51.010980 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.010954    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-139.ec2.internal"
Jul 28 12:33:51.016704 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.016689    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-139.ec2.internal"
Jul 28 12:33:51.016796 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:51.016710    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-139.ec2.internal\": node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:51.018939 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.018922    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 12:33:51.019018 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.018953    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 12:33:51.019018 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.018974    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 12:33:51.019018 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:51.019008    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 12:33:51.021034 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.021019    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:51.054802 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:51.054747    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:51.119822 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.119793    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal"]
Jul 28 12:33:51.119976 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.119869    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:51.121705 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.121688    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:51.121779 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.121718    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:51.121779 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.121729    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:51.122914 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.122901    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:51.123053 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.123037    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:51.123117 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.123072    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:51.123641 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.123626    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:51.123727 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.123642    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:51.123727 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.123651    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:51.123727 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.123663    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:51.123727 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.123667    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:51.123727 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.123674    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:51.124959 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.124941    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:51.125042 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.124975    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:33:51.126110 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.126095    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:33:51.126191 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.126121    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:33:51.126191 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.126134    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:33:51.152793 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.152766    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19f6277a0055c64139d8e6ee0d720a13-config\") pod \"kube-apiserver-proxy-ip-10-0-142-139.ec2.internal\" (UID: \"19f6277a0055c64139d8e6ee0d720a13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:51.152793 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.152793    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a42885fda7bccde2b2c5aa772ed768e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"a42885fda7bccde2b2c5aa772ed768e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:51.152978 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:51.152811    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a42885fda7bccde2b2c5aa772ed768e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"a42885fda7bccde2b2c5aa772ed768e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:51.154225 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:51.154206    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-139.ec2.internal\" not found" node="ip-10-0-142-139.ec2.internal"
Jul 28 12:33:51.154909 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:51.154892    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:51.158618 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:51.158603    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-139.ec2.internal\" not found" node="ip-10-0-142-139.ec2.internal"
Jul 28 12:33:52.480338 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:52.480311    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a42885fda7bccde2b2c5aa772ed768e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"a42885fda7bccde2b2c5aa772ed768e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:52.480338 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:52.480340    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19f6277a0055c64139d8e6ee0d720a13-config\") pod \"kube-apiserver-proxy-ip-10-0-142-139.ec2.internal\" (UID: \"19f6277a0055c64139d8e6ee0d720a13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:52.480513 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:52.480355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a42885fda7bccde2b2c5aa772ed768e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"a42885fda7bccde2b2c5aa772ed768e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:52.480513 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:52.480390    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a42885fda7bccde2b2c5aa772ed768e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"a42885fda7bccde2b2c5aa772ed768e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:52.480513 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:52.480423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19f6277a0055c64139d8e6ee0d720a13-config\") pod \"kube-apiserver-proxy-ip-10-0-142-139.ec2.internal\" (UID: \"19f6277a0055c64139d8e6ee0d720a13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:52.480513 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:52.480394    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a42885fda7bccde2b2c5aa772ed768e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"a42885fda7bccde2b2c5aa772ed768e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:52.482424 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:52.482411    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:52.583333 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:52.583278    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:52.682755 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:52.682717    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:52.683812 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:52.683796    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:52.688313 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:52.688082    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:52.784116 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:52.784084    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:52.884626 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:52.884552    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:52.985132 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:52.985104    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found"
Jul 28 12:33:53.025306 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.025274    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 12:33:53.025718 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.025434    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:33:53.080779 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.080752    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:53.094113 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.094082    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:28:52 +0000 UTC" deadline="2028-01-23 08:25:28.009985151 +0000 UTC"
Jul 28 12:33:53.094113 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.094109    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13051h51m34.915879439s"
Jul 28 12:33:53.177426 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.177403    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:53.190462 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.190441    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:33:53.191903 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.191888    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal"
Jul 28 12:33:53.200129 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.200110    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:33:53.252713 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:53.252688    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f6277a0055c64139d8e6ee0d720a13.slice/crio-28422a9b37a15a6061397a18e3ffb02e331fbc0244ac86129f7e4ef2b8123ace WatchSource:0}: Error finding container 28422a9b37a15a6061397a18e3ffb02e331fbc0244ac86129f7e4ef2b8123ace: Status 404 returned error can't find the container with id 28422a9b37a15a6061397a18e3ffb02e331fbc0244ac86129f7e4ef2b8123ace
Jul 28 12:33:53.253592 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:53.253571    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42885fda7bccde2b2c5aa772ed768e0.slice/crio-45e7ff85ac925a1c268a0646ac202daeb8224c00d8e01233835f616ad222861b WatchSource:0}: Error finding container 45e7ff85ac925a1c268a0646ac202daeb8224c00d8e01233835f616ad222861b: Status 404 returned error can't find the container with id 45e7ff85ac925a1c268a0646ac202daeb8224c00d8e01233835f616ad222861b
Jul 28 12:33:53.258353 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.258338    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:33:53.367302 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.367275    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:53.582621 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:53.582598    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:54.060749 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.060721    2569 apiserver.go:52] "Watching apiserver"
Jul 28 12:33:54.066767 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.066602    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:33:54.070024 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.069845    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 12:33:54.070869 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.070844    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r","openshift-dns/node-resolver-zkstz","openshift-image-registry/node-ca-4vq25","openshift-multus/multus-7ng8z","openshift-multus/multus-additional-cni-plugins-pn8wq","openshift-network-diagnostics/network-check-target-kffxs","openshift-ovn-kubernetes/ovnkube-node-45gqf","kube-system/global-pull-secret-syncer-xdsfj","kube-system/konnectivity-agent-lcl5q","openshift-cluster-node-tuning-operator/tuned-kr4z5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal","openshift-multus/network-metrics-daemon-6jp6n","openshift-network-operator/iptables-alerter-wjhph"]
Jul 28 12:33:54.073145 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.072813    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:33:54.073145 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.072894    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:33:54.074560 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.074533    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:54.074663 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.074618    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:33:54.076447 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.076427    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.076976 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.076898    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 12:33:54.078221 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.078060    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.078976 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.078612    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c89hh\""
Jul 28 12:33:54.078976 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.078832    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 12:33:54.078976 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.078859    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 12:33:54.080186 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.080089    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 12:33:54.080495 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.080350    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vswcr\""
Jul 28 12:33:54.080495 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.080394    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 12:33:54.080962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.080800    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 12:33:54.080962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.080893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.084542 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.083753    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 12:33:54.084542 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.083805    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 12:33:54.084542 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.084016    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 12:33:54.084542 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.084023    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 12:33:54.084542 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.084223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.084542 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.084242    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wvr8d\""
Jul 28 12:33:54.088066 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.088046    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:54.088186 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.088148    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:33:54.088492 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.088442    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 12:33:54.088542 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.088512    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-99pgh\""
Jul 28 12:33:54.088844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.088827    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 12:33:54.089945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.089172    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c599f721-e2df-422a-bf36-6319e85593d2-host\") pod \"node-ca-4vq25\" (UID: \"c599f721-e2df-422a-bf36-6319e85593d2\") " pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.089945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.089215    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c599f721-e2df-422a-bf36-6319e85593d2-serviceca\") pod \"node-ca-4vq25\" (UID: \"c599f721-e2df-422a-bf36-6319e85593d2\") " pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.089945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.089227    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.089945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.089244    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhm4p\" (UniqueName: \"kubernetes.io/projected/c599f721-e2df-422a-bf36-6319e85593d2-kube-api-access-rhm4p\") pod \"node-ca-4vq25\" (UID: \"c599f721-e2df-422a-bf36-6319e85593d2\") " pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.089945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.089280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh48c\" (UniqueName: \"kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c\") pod \"network-check-target-kffxs\" (UID: \"a4544365-5759-41c2-8b2a-0f96b508e3d5\") " pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:33:54.089945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.089312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxsth\" (UniqueName: \"kubernetes.io/projected/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-kube-api-access-mxsth\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:54.089945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.089346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/793c18e4-9309-4da3-956e-5791d83891cf-hosts-file\") pod \"node-resolver-zkstz\" (UID: \"793c18e4-9309-4da3-956e-5791d83891cf\") " pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.089945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.089374    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhlnv\" (UniqueName: \"kubernetes.io/projected/793c18e4-9309-4da3-956e-5791d83891cf-kube-api-access-zhlnv\") pod \"node-resolver-zkstz\" (UID: \"793c18e4-9309-4da3-956e-5791d83891cf\") " pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.089945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.089401    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:54.089945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.089448    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793c18e4-9309-4da3-956e-5791d83891cf-tmp-dir\") pod \"node-resolver-zkstz\" (UID: \"793c18e4-9309-4da3-956e-5791d83891cf\") " pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.092046 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.091528    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.093225 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.092808    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:33:54.093225 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.092923    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 12:33:54.093225 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.092942    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 12:33:54.093225 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.093077    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wcbx6\""
Jul 28 12:33:54.094743 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.093784    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bp6n6\""
Jul 28 12:33:54.094743 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.094025    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 12:33:54.094743 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.094333    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:33:54.094743 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.094380    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 12:33:54.094743 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.094492    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 12:33:54.096111 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.096091    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.097171 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.096834    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 12:33:54.097171 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.096908    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 12:33:54.097171 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.096842    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ft277\""
Jul 28 12:33:54.097775 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.097756    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:33:54.098407 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.098384    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.098739 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.098723    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:33:54.099021 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.098736    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bklq6\""
Jul 28 12:33:54.099186 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.098954    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 12:33:54.100726 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.100708    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 12:33:54.101523 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.101421    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 12:33:54.101866 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.101846    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 12:33:54.101964 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.101946    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 12:33:54.102056 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.101858    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 12:33:54.102138 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.102122    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fm9ds\""
Jul 28 12:33:54.102255 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.102242    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 12:33:54.113046 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.112983    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9xdsm"
Jul 28 12:33:54.123542 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.123456    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9xdsm"
Jul 28 12:33:54.178511 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.178484    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 12:33:54.190717 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190475    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-cnibin\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.190717 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190517    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-run-k8s-cni-cncf-io\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.190717 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190549    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-sysconfig\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.190717 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190570    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-systemd\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.190717 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190597    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjrbj\" (UniqueName: \"kubernetes.io/projected/0e25259e-78a2-4063-8946-e80c2954e647-kube-api-access-sjrbj\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.190717 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190634    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-var-lib-openvswitch\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.190717 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190662    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793c18e4-9309-4da3-956e-5791d83891cf-tmp-dir\") pod \"node-resolver-zkstz\" (UID: \"793c18e4-9309-4da3-956e-5791d83891cf\") " pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.190717 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-var-lib-cni-bin\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-var-lib-cni-multus\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5prs\" (UniqueName: \"kubernetes.io/projected/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-kube-api-access-s5prs\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79d7959e-9636-495a-930c-a09f71fb6c57-kubelet-config\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190854    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79d7959e-9636-495a-930c-a09f71fb6c57-dbus\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190896    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/793c18e4-9309-4da3-956e-5791d83891cf-hosts-file\") pod \"node-resolver-zkstz\" (UID: \"793c18e4-9309-4da3-956e-5791d83891cf\") " pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.190921    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-device-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69344ace-2af3-4833-bc5f-cc6e74c00933-ovnkube-config\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191028    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793c18e4-9309-4da3-956e-5791d83891cf-tmp-dir\") pod \"node-resolver-zkstz\" (UID: \"793c18e4-9309-4da3-956e-5791d83891cf\") " pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191030    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-cni-dir\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-var-lib-kubelet\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191083    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-host\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-run-systemd\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191138    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/793c18e4-9309-4da3-956e-5791d83891cf-hosts-file\") pod \"node-resolver-zkstz\" (UID: \"793c18e4-9309-4da3-956e-5791d83891cf\") " pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh48c\" (UniqueName: \"kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c\") pod \"network-check-target-kffxs\" (UID: \"a4544365-5759-41c2-8b2a-0f96b508e3d5\") " pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191191    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-run-netns\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191229    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-run-multus-certs\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.191273 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191265    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-etc-kubernetes\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191370    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-kubelet\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191394    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnnc\" (UniqueName: \"kubernetes.io/projected/69344ace-2af3-4833-bc5f-cc6e74c00933-kube-api-access-xpnnc\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191440    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-system-cni-dir\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191476    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-sys-fs\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191526    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-run-netns\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191546    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-etc-openvswitch\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191569    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-cni-bin\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191616    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c599f721-e2df-422a-bf36-6319e85593d2-host\") pod \"node-ca-4vq25\" (UID: \"c599f721-e2df-422a-bf36-6319e85593d2\") " pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191641    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c599f721-e2df-422a-bf36-6319e85593d2-serviceca\") pod \"node-ca-4vq25\" (UID: \"c599f721-e2df-422a-bf36-6319e85593d2\") " pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191673    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-modprobe-d\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c599f721-e2df-422a-bf36-6319e85593d2-host\") pod \"node-ca-4vq25\" (UID: \"c599f721-e2df-422a-bf36-6319e85593d2\") " pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191812    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8hg\" (UniqueName: \"kubernetes.io/projected/46cbac40-95c4-4172-b7d9-ffe38426e007-kube-api-access-xb8hg\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69344ace-2af3-4833-bc5f-cc6e74c00933-ovnkube-script-lib\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191866    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-tmp\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191924    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.192050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.191968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9de0429-8e9d-4263-8d35-b8088687e3cc-cni-binary-copy\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192009    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-kubernetes\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192058    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-run\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192080    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-socket-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhm4p\" (UniqueName: \"kubernetes.io/projected/c599f721-e2df-422a-bf36-6319e85593d2-kube-api-access-rhm4p\") pod \"node-ca-4vq25\" (UID: \"c599f721-e2df-422a-bf36-6319e85593d2\") " pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192161    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-hostroot\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c599f721-e2df-422a-bf36-6319e85593d2-serviceca\") pod \"node-ca-4vq25\" (UID: \"c599f721-e2df-422a-bf36-6319e85593d2\") " pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192185    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-systemd-units\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192221    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-run-ovn\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192244    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-cni-netd\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192287    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-os-release\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192341    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e25259e-78a2-4063-8946-e80c2954e647-cni-binary-copy\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.192836 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w52wd\" (UniqueName: \"kubernetes.io/projected/ae3368dc-9e5f-44c0-8f49-532e8336cbb7-kube-api-access-w52wd\") pod \"iptables-alerter-wjhph\" (UID: \"ae3368dc-9e5f-44c0-8f49-532e8336cbb7\") " pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-os-release\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-conf-dir\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192891    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae3368dc-9e5f-44c0-8f49-532e8336cbb7-iptables-alerter-script\") pod \"iptables-alerter-wjhph\" (UID: \"ae3368dc-9e5f-44c0-8f49-532e8336cbb7\") " pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192915    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-slash\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-tuned\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.192980    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-system-cni-dir\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e25259e-78a2-4063-8946-e80c2954e647-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193041    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e25259e-78a2-4063-8946-e80c2954e647-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193083    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-run-openvswitch\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193103    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69344ace-2af3-4833-bc5f-cc6e74c00933-ovn-node-metrics-cert\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhlnv\" (UniqueName: \"kubernetes.io/projected/793c18e4-9309-4da3-956e-5791d83891cf-kube-api-access-zhlnv\") pod \"node-resolver-zkstz\" (UID: \"793c18e4-9309-4da3-956e-5791d83891cf\") " pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193160    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4ae7f-5807-4215-8fc4-a708450f9777-konnectivity-ca\") pod \"konnectivity-agent-lcl5q\" (UID: \"fbe4ae7f-5807-4215-8fc4-a708450f9777\") " pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-sys\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193215    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-cnibin\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.193476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193229    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae3368dc-9e5f-44c0-8f49-532e8336cbb7-host-slash\") pod \"iptables-alerter-wjhph\" (UID: \"ae3368dc-9e5f-44c0-8f49-532e8336cbb7\") " pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193244    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193265    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-log-socket\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69344ace-2af3-4833-bc5f-cc6e74c00933-env-overrides\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-socket-dir-parent\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193320    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-sysctl-conf\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193387    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxsth\" (UniqueName: \"kubernetes.io/projected/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-kube-api-access-mxsth\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193403    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-var-lib-kubelet\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193417    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-daemon-config\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193437    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-lib-modules\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193823    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193880    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxb6\" (UniqueName: \"kubernetes.io/projected/b9de0429-8e9d-4263-8d35-b8088687e3cc-kube-api-access-9pxb6\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193923    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fbe4ae7f-5807-4215-8fc4-a708450f9777-agent-certs\") pod \"konnectivity-agent-lcl5q\" (UID: \"fbe4ae7f-5807-4215-8fc4-a708450f9777\") " pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193957    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-sysctl-d\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.193979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-registration-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.194017    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-node-log\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.194038    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-run-ovn-kubernetes\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.194233 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.194180    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:54.195026 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.194241    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs podName:afc8521c-7a71-4ec0-9e3a-53d4512d5ae3 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:54.694222322 +0000 UTC m=+2.909738967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs") pod "network-metrics-daemon-6jp6n" (UID: "afc8521c-7a71-4ec0-9e3a-53d4512d5ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:54.201545 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.201522    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:33:54.201674 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.201550    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:33:54.201674 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.201564    2569 projected.go:196] Error preparing data for projected volume kube-api-access-bh48c for pod openshift-network-diagnostics/network-check-target-kffxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:54.201674 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.201624    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c podName:a4544365-5759-41c2-8b2a-0f96b508e3d5 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:54.701605956 +0000 UTC m=+2.917122603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bh48c" (UniqueName: "kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c") pod "network-check-target-kffxs" (UID: "a4544365-5759-41c2-8b2a-0f96b508e3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:54.202115 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.202079    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 12:33:54.206316 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.206282    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhm4p\" (UniqueName: \"kubernetes.io/projected/c599f721-e2df-422a-bf36-6319e85593d2-kube-api-access-rhm4p\") pod \"node-ca-4vq25\" (UID: \"c599f721-e2df-422a-bf36-6319e85593d2\") " pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.206896 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.206874    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhlnv\" (UniqueName: \"kubernetes.io/projected/793c18e4-9309-4da3-956e-5791d83891cf-kube-api-access-zhlnv\") pod \"node-resolver-zkstz\" (UID: \"793c18e4-9309-4da3-956e-5791d83891cf\") " pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.212016 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.211977    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxsth\" (UniqueName: \"kubernetes.io/projected/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-kube-api-access-mxsth\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:54.250319 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.250267    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" event={"ID":"a42885fda7bccde2b2c5aa772ed768e0","Type":"ContainerStarted","Data":"45e7ff85ac925a1c268a0646ac202daeb8224c00d8e01233835f616ad222861b"}
Jul 28 12:33:54.251650 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.251620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" event={"ID":"19f6277a0055c64139d8e6ee0d720a13","Type":"ContainerStarted","Data":"28422a9b37a15a6061397a18e3ffb02e331fbc0244ac86129f7e4ef2b8123ace"}
Jul 28 12:33:54.294530 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294502    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae3368dc-9e5f-44c0-8f49-532e8336cbb7-host-slash\") pod \"iptables-alerter-wjhph\" (UID: \"ae3368dc-9e5f-44c0-8f49-532e8336cbb7\") " pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.294699 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294537    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.294699 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294558    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-log-socket\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.294699 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69344ace-2af3-4833-bc5f-cc6e74c00933-env-overrides\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.294699 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-socket-dir-parent\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.294699 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-sysctl-conf\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.294699 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294634    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae3368dc-9e5f-44c0-8f49-532e8336cbb7-host-slash\") pod \"iptables-alerter-wjhph\" (UID: \"ae3368dc-9e5f-44c0-8f49-532e8336cbb7\") " pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.294699 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-var-lib-kubelet\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294742    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-log-socket\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294694    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-var-lib-kubelet\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-daemon-config\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294784    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-sysctl-conf\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294803    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-socket-dir-parent\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-lib-modules\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294863    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxb6\" (UniqueName: \"kubernetes.io/projected/b9de0429-8e9d-4263-8d35-b8088687e3cc-kube-api-access-9pxb6\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294889    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fbe4ae7f-5807-4215-8fc4-a708450f9777-agent-certs\") pod \"konnectivity-agent-lcl5q\" (UID: \"fbe4ae7f-5807-4215-8fc4-a708450f9777\") " pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294916    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-sysctl-d\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-lib-modules\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-registration-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.294964    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-node-log\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.295023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-run-ovn-kubernetes\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295035    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-cnibin\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295056    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-run-k8s-cni-cncf-io\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295079    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-sysconfig\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295072    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-sysctl-d\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295102    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-systemd\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295125    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjrbj\" (UniqueName: \"kubernetes.io/projected/0e25259e-78a2-4063-8946-e80c2954e647-kube-api-access-sjrbj\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295137    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-run-ovn-kubernetes\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-var-lib-openvswitch\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295163    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-registration-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295172    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-var-lib-cni-bin\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295183    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-node-log\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-var-lib-cni-multus\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295199    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69344ace-2af3-4833-bc5f-cc6e74c00933-env-overrides\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295214    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-sysconfig\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5prs\" (UniqueName: \"kubernetes.io/projected/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-kube-api-access-s5prs\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295229    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-daemon-config\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.295648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295237    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-cnibin\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79d7959e-9636-495a-930c-a09f71fb6c57-kubelet-config\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295283    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-systemd\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79d7959e-9636-495a-930c-a09f71fb6c57-kubelet-config\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295365    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-var-lib-openvswitch\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295404    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-var-lib-cni-bin\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295445    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-run-k8s-cni-cncf-io\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295495    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-var-lib-cni-multus\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79d7959e-9636-495a-930c-a09f71fb6c57-dbus\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295534    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-device-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295553    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69344ace-2af3-4833-bc5f-cc6e74c00933-ovnkube-config\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295576    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-cni-dir\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295593    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-device-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-var-lib-kubelet\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-var-lib-kubelet\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295658    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79d7959e-9636-495a-930c-a09f71fb6c57-dbus\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295666    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-host\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295668    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-cni-dir\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.296427 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295701    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-host\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295694    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-run-systemd\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295727    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-run-systemd\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295755    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-run-netns\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-run-multus-certs\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295805    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-etc-kubernetes\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295829    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-kubelet\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295842    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-run-netns\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnnc\" (UniqueName: \"kubernetes.io/projected/69344ace-2af3-4833-bc5f-cc6e74c00933-kube-api-access-xpnnc\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295878    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-etc-kubernetes\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295882    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-kubelet\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295886    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-system-cni-dir\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-host-run-multus-certs\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295919    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295946    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-sys-fs\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295962    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-system-cni-dir\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.295970    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-run-netns\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296015    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-etc-openvswitch\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.297333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296039    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-cni-bin\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.296059    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-modprobe-d\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296091    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8hg\" (UniqueName: \"kubernetes.io/projected/46cbac40-95c4-4172-b7d9-ffe38426e007-kube-api-access-xb8hg\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296094    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-run-netns\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296061    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69344ace-2af3-4833-bc5f-cc6e74c00933-ovnkube-config\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.296112    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret podName:79d7959e-9636-495a-930c-a09f71fb6c57 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:54.796096327 +0000 UTC m=+3.011612960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret") pod "global-pull-secret-syncer-xdsfj" (UID: "79d7959e-9636-495a-930c-a09f71fb6c57") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-sys-fs\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296163    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-etc-openvswitch\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296184    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69344ace-2af3-4833-bc5f-cc6e74c00933-ovnkube-script-lib\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-tmp\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296233    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9de0429-8e9d-4263-8d35-b8088687e3cc-cni-binary-copy\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-cni-bin\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-modprobe-d\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-kubernetes\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-run\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.298122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296329    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-socket-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296352    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-hostroot\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-systemd-units\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296396    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-run-ovn\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296403    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296419    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-cni-netd\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-os-release\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296482    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-socket-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296505    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e25259e-78a2-4063-8946-e80c2954e647-cni-binary-copy\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w52wd\" (UniqueName: \"kubernetes.io/projected/ae3368dc-9e5f-44c0-8f49-532e8336cbb7-kube-api-access-w52wd\") pod \"iptables-alerter-wjhph\" (UID: \"ae3368dc-9e5f-44c0-8f49-532e8336cbb7\") " pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-os-release\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-os-release\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-conf-dir\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296596    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-os-release\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae3368dc-9e5f-44c0-8f49-532e8336cbb7-iptables-alerter-script\") pod \"iptables-alerter-wjhph\" (UID: \"ae3368dc-9e5f-44c0-8f49-532e8336cbb7\") " pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296630    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.299003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296685    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-multus-conf-dir\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296719    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-run-ovn\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296727    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46cbac40-95c4-4172-b7d9-ffe38426e007-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296743    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-run\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9de0429-8e9d-4263-8d35-b8088687e3cc-cni-binary-copy\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296762    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-cni-netd\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-slash\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296792    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-kubernetes\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296808    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-host-slash\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296833    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9de0429-8e9d-4263-8d35-b8088687e3cc-hostroot\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296848    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-systemd-units\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296879    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-tuned\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296905    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-system-cni-dir\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e25259e-78a2-4063-8946-e80c2954e647-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e25259e-78a2-4063-8946-e80c2954e647-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296978    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-system-cni-dir\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.299844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.296981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-run-openvswitch\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69344ace-2af3-4833-bc5f-cc6e74c00933-ovn-node-metrics-cert\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297101    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4ae7f-5807-4215-8fc4-a708450f9777-konnectivity-ca\") pod \"konnectivity-agent-lcl5q\" (UID: \"fbe4ae7f-5807-4215-8fc4-a708450f9777\") " pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-sys\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297154    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-cnibin\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297220    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e25259e-78a2-4063-8946-e80c2954e647-cnibin\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297220    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae3368dc-9e5f-44c0-8f49-532e8336cbb7-iptables-alerter-script\") pod \"iptables-alerter-wjhph\" (UID: \"ae3368dc-9e5f-44c0-8f49-532e8336cbb7\") " pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297102    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69344ace-2af3-4833-bc5f-cc6e74c00933-ovnkube-script-lib\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297265    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-sys\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297265    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e25259e-78a2-4063-8946-e80c2954e647-cni-binary-copy\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297321    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69344ace-2af3-4833-bc5f-cc6e74c00933-run-openvswitch\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297443    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e25259e-78a2-4063-8946-e80c2954e647-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.297778    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e25259e-78a2-4063-8946-e80c2954e647-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.298246    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fbe4ae7f-5807-4215-8fc4-a708450f9777-agent-certs\") pod \"konnectivity-agent-lcl5q\" (UID: \"fbe4ae7f-5807-4215-8fc4-a708450f9777\") " pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.299204    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-etc-tuned\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.299371    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-tmp\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.300706 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.299393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69344ace-2af3-4833-bc5f-cc6e74c00933-ovn-node-metrics-cert\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.306165 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.306145    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fbe4ae7f-5807-4215-8fc4-a708450f9777-konnectivity-ca\") pod \"konnectivity-agent-lcl5q\" (UID: \"fbe4ae7f-5807-4215-8fc4-a708450f9777\") " pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:33:54.311963 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.311829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxb6\" (UniqueName: \"kubernetes.io/projected/b9de0429-8e9d-4263-8d35-b8088687e3cc-kube-api-access-9pxb6\") pod \"multus-7ng8z\" (UID: \"b9de0429-8e9d-4263-8d35-b8088687e3cc\") " pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.312076 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.312003    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8hg\" (UniqueName: \"kubernetes.io/projected/46cbac40-95c4-4172-b7d9-ffe38426e007-kube-api-access-xb8hg\") pod \"aws-ebs-csi-driver-node-wqp5r\" (UID: \"46cbac40-95c4-4172-b7d9-ffe38426e007\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.312281 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.312264    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjrbj\" (UniqueName: \"kubernetes.io/projected/0e25259e-78a2-4063-8946-e80c2954e647-kube-api-access-sjrbj\") pod \"multus-additional-cni-plugins-pn8wq\" (UID: \"0e25259e-78a2-4063-8946-e80c2954e647\") " pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.312442 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.312427    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5prs\" (UniqueName: \"kubernetes.io/projected/c533d0e1-4c5f-4cb4-8259-b83b72a39e1d-kube-api-access-s5prs\") pod \"tuned-kr4z5\" (UID: \"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d\") " pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.312727 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.312704    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnnc\" (UniqueName: \"kubernetes.io/projected/69344ace-2af3-4833-bc5f-cc6e74c00933-kube-api-access-xpnnc\") pod \"ovnkube-node-45gqf\" (UID: \"69344ace-2af3-4833-bc5f-cc6e74c00933\") " pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.312813 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.312763    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w52wd\" (UniqueName: \"kubernetes.io/projected/ae3368dc-9e5f-44c0-8f49-532e8336cbb7-kube-api-access-w52wd\") pod \"iptables-alerter-wjhph\" (UID: \"ae3368dc-9e5f-44c0-8f49-532e8336cbb7\") " pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.394855 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.394824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zkstz"
Jul 28 12:33:54.403072 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.403048    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4vq25"
Jul 28 12:33:54.412670 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.412651    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:33:54.418313 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.418291    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7ng8z"
Jul 28 12:33:54.427072 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.427044    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pn8wq"
Jul 28 12:33:54.435671 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.435648    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wjhph"
Jul 28 12:33:54.442319 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.442298    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r"
Jul 28 12:33:54.454935 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.454905    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kr4z5"
Jul 28 12:33:54.461803 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.461777    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:33:54.700402 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.700329    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:54.700572 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.700464    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:54.700572 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.700526    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs podName:afc8521c-7a71-4ec0-9e3a-53d4512d5ae3 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:55.700508284 +0000 UTC m=+3.916024914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs") pod "network-metrics-daemon-6jp6n" (UID: "afc8521c-7a71-4ec0-9e3a-53d4512d5ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:54.800700 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.800664    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh48c\" (UniqueName: \"kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c\") pod \"network-check-target-kffxs\" (UID: \"a4544365-5759-41c2-8b2a-0f96b508e3d5\") " pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:33:54.800885 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:54.800712    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:54.800885 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.800824    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:54.800885 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.800843    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:33:54.800885 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.800871    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:33:54.800885 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.800884    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret podName:79d7959e-9636-495a-930c-a09f71fb6c57 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:55.800869138 +0000 UTC m=+4.016385789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret") pod "global-pull-secret-syncer-xdsfj" (UID: "79d7959e-9636-495a-930c-a09f71fb6c57") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:54.800885 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.800886    2569 projected.go:196] Error preparing data for projected volume kube-api-access-bh48c for pod openshift-network-diagnostics/network-check-target-kffxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:54.801222 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:54.800939    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c podName:a4544365-5759-41c2-8b2a-0f96b508e3d5 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:55.800918982 +0000 UTC m=+4.016435614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bh48c" (UniqueName: "kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c") pod "network-check-target-kffxs" (UID: "a4544365-5759-41c2-8b2a-0f96b508e3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:55.126418 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.126390    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:28:54 +0000 UTC" deadline="2028-02-14 03:26:37.605825805 +0000 UTC"
Jul 28 12:33:55.126855 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.126419    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13574h52m42.479414887s"
Jul 28 12:33:55.155256 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:55.155229    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9de0429_8e9d_4263_8d35_b8088687e3cc.slice/crio-d3aca81d60c8649bf6974ce0b201f8381b625d397ed46438a0a78b2631a1fcdb WatchSource:0}: Error finding container d3aca81d60c8649bf6974ce0b201f8381b625d397ed46438a0a78b2631a1fcdb: Status 404 returned error can't find the container with id d3aca81d60c8649bf6974ce0b201f8381b625d397ed46438a0a78b2631a1fcdb
Jul 28 12:33:55.155785 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:55.155756    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cbac40_95c4_4172_b7d9_ffe38426e007.slice/crio-f2f41faf336a22f5e3c89b858814e75adf06d743b74355e7aed8e7a686436ae9 WatchSource:0}: Error finding container f2f41faf336a22f5e3c89b858814e75adf06d743b74355e7aed8e7a686436ae9: Status 404 returned error can't find the container with id f2f41faf336a22f5e3c89b858814e75adf06d743b74355e7aed8e7a686436ae9
Jul 28 12:33:55.177836 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:55.177812    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc533d0e1_4c5f_4cb4_8259_b83b72a39e1d.slice/crio-7cb1eced152854e1eb353339ba9b520e44d3c756143248e63463322eab1df191 WatchSource:0}: Error finding container 7cb1eced152854e1eb353339ba9b520e44d3c756143248e63463322eab1df191: Status 404 returned error can't find the container with id 7cb1eced152854e1eb353339ba9b520e44d3c756143248e63463322eab1df191
Jul 28 12:33:55.178699 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:33:55.178677    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e25259e_78a2_4063_8946_e80c2954e647.slice/crio-1f06769463ad563d8f39a09b6614a3c6dca8d00bd4ba28afb4a44c0937fd73be WatchSource:0}: Error finding container 1f06769463ad563d8f39a09b6614a3c6dca8d00bd4ba28afb4a44c0937fd73be: Status 404 returned error can't find the container with id 1f06769463ad563d8f39a09b6614a3c6dca8d00bd4ba28afb4a44c0937fd73be
Jul 28 12:33:55.246053 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.245888    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:55.246190 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:55.246166    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:33:55.255166 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.255137    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zkstz" event={"ID":"793c18e4-9309-4da3-956e-5791d83891cf","Type":"ContainerStarted","Data":"5d961c203c9adf352055e6fc3ef42dbcd6b44af56cdf22cd2cfe0821dd2d6326"}
Jul 28 12:33:55.256250 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.256224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lcl5q" event={"ID":"fbe4ae7f-5807-4215-8fc4-a708450f9777","Type":"ContainerStarted","Data":"29631853dc7b58a8cb8274432d420751b4c1529d0199ebbf63cb017f15e0f85d"}
Jul 28 12:33:55.257362 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.257333    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r" event={"ID":"46cbac40-95c4-4172-b7d9-ffe38426e007","Type":"ContainerStarted","Data":"f2f41faf336a22f5e3c89b858814e75adf06d743b74355e7aed8e7a686436ae9"}
Jul 28 12:33:55.258497 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.258474    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4vq25" event={"ID":"c599f721-e2df-422a-bf36-6319e85593d2","Type":"ContainerStarted","Data":"ce0c769cbebf71449efece8b9b40c802b4bc64e5a8ebe36dba5b1ff25e050daa"}
Jul 28 12:33:55.259578 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.259560    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn8wq" event={"ID":"0e25259e-78a2-4063-8946-e80c2954e647","Type":"ContainerStarted","Data":"1f06769463ad563d8f39a09b6614a3c6dca8d00bd4ba28afb4a44c0937fd73be"}
Jul 28 12:33:55.260962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.260940    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kr4z5" event={"ID":"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d","Type":"ContainerStarted","Data":"7cb1eced152854e1eb353339ba9b520e44d3c756143248e63463322eab1df191"}
Jul 28 12:33:55.262050 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.262018    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7ng8z" event={"ID":"b9de0429-8e9d-4263-8d35-b8088687e3cc","Type":"ContainerStarted","Data":"d3aca81d60c8649bf6974ce0b201f8381b625d397ed46438a0a78b2631a1fcdb"}
Jul 28 12:33:55.263393 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.263370    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wjhph" event={"ID":"ae3368dc-9e5f-44c0-8f49-532e8336cbb7","Type":"ContainerStarted","Data":"76d2c359357bea1629220f636210e51b31867bcd338d4bca70de1270f81e7648"}
Jul 28 12:33:55.264742 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.264722    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf" event={"ID":"69344ace-2af3-4833-bc5f-cc6e74c00933","Type":"ContainerStarted","Data":"314fbd46e25fff505df9761ccaeae94274fb890c52fc3408a56c97f3114e8ea9"}
Jul 28 12:33:55.707592 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.707510    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:55.707744 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:55.707709    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:55.707808 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:55.707772    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs podName:afc8521c-7a71-4ec0-9e3a-53d4512d5ae3 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:57.707754164 +0000 UTC m=+5.923270809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs") pod "network-metrics-daemon-6jp6n" (UID: "afc8521c-7a71-4ec0-9e3a-53d4512d5ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:55.808472 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.808428    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh48c\" (UniqueName: \"kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c\") pod \"network-check-target-kffxs\" (UID: \"a4544365-5759-41c2-8b2a-0f96b508e3d5\") " pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:33:55.808655 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:55.808485    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:55.808655 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:55.808646    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:55.808764 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:55.808708    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret podName:79d7959e-9636-495a-930c-a09f71fb6c57 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:57.808690667 +0000 UTC m=+6.024207310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret") pod "global-pull-secret-syncer-xdsfj" (UID: "79d7959e-9636-495a-930c-a09f71fb6c57") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:55.809215 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:55.809194    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:33:55.809215 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:55.809219    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:33:55.809376 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:55.809232    2569 projected.go:196] Error preparing data for projected volume kube-api-access-bh48c for pod openshift-network-diagnostics/network-check-target-kffxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:55.809376 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:55.809279    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c podName:a4544365-5759-41c2-8b2a-0f96b508e3d5 nodeName:}" failed. No retries permitted until 2026-07-28 12:33:57.80926358 +0000 UTC m=+6.024780231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bh48c" (UniqueName: "kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c") pod "network-check-target-kffxs" (UID: "a4544365-5759-41c2-8b2a-0f96b508e3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:56.127145 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:56.127080    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:28:54 +0000 UTC" deadline="2028-05-03 17:38:15.013193604 +0000 UTC"
Jul 28 12:33:56.127145 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:56.127143    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15485h4m18.886054496s"
Jul 28 12:33:56.248861 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:56.248178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:33:56.248861 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:56.248252    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:56.248861 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:56.248356    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:33:56.248861 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:56.248757    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:33:56.292614 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:56.292574    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" event={"ID":"19f6277a0055c64139d8e6ee0d720a13","Type":"ContainerStarted","Data":"f86cb278f66fbe9dad3fa2a6cb44e9f4512993698a84fea867cdcf49edfcddcd"}
Jul 28 12:33:56.313559 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:56.313521    2569 generic.go:356] "Generic (PLEG): container finished" podID="a42885fda7bccde2b2c5aa772ed768e0" containerID="9110dacd731eca7ac6376008ecf20c3774d304e992e478dcd8c53a665dbf301d" exitCode=0
Jul 28 12:33:56.313559 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:56.313563    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" event={"ID":"a42885fda7bccde2b2c5aa772ed768e0","Type":"ContainerDied","Data":"9110dacd731eca7ac6376008ecf20c3774d304e992e478dcd8c53a665dbf301d"}
Jul 28 12:33:56.337869 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:56.337815    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" podStartSLOduration=3.337794069 podStartE2EDuration="3.337794069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:56.321667618 +0000 UTC m=+4.537184272" watchObservedRunningTime="2026-07-28 12:33:56.337794069 +0000 UTC m=+4.553310724"
Jul 28 12:33:57.246870 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:57.246374    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:57.246870 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:57.246516    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:33:57.332098 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:57.332034    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" event={"ID":"a42885fda7bccde2b2c5aa772ed768e0","Type":"ContainerStarted","Data":"0a0a74777318ced139cafdbe5ae3825fc4d0d6b13f3b22dc23a5baa63039f496"}
Jul 28 12:33:57.724100 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:57.724059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:57.724292 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:57.724281    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:57.724360 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:57.724343    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs podName:afc8521c-7a71-4ec0-9e3a-53d4512d5ae3 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:01.724324023 +0000 UTC m=+9.939840660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs") pod "network-metrics-daemon-6jp6n" (UID: "afc8521c-7a71-4ec0-9e3a-53d4512d5ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:33:57.824664 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:57.824625    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh48c\" (UniqueName: \"kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c\") pod \"network-check-target-kffxs\" (UID: \"a4544365-5759-41c2-8b2a-0f96b508e3d5\") " pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:33:57.824848 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:57.824684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:57.824918 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:57.824844    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:57.824918 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:57.824904    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret podName:79d7959e-9636-495a-930c-a09f71fb6c57 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:01.824887862 +0000 UTC m=+10.040404496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret") pod "global-pull-secret-syncer-xdsfj" (UID: "79d7959e-9636-495a-930c-a09f71fb6c57") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:33:57.825360 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:57.825335    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:33:57.825360 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:57.825357    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:33:57.825508 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:57.825370    2569 projected.go:196] Error preparing data for projected volume kube-api-access-bh48c for pod openshift-network-diagnostics/network-check-target-kffxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:57.825508 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:57.825414    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c podName:a4544365-5759-41c2-8b2a-0f96b508e3d5 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:01.82539975 +0000 UTC m=+10.040916385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bh48c" (UniqueName: "kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c") pod "network-check-target-kffxs" (UID: "a4544365-5759-41c2-8b2a-0f96b508e3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:33:58.250511 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:58.250420    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:33:58.250953 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:58.250623    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:33:58.251145 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:58.251126    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:33:58.251266 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:58.251237    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:33:59.246215 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:33:59.246180    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:33:59.246394 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:33:59.246325    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:00.246491 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:00.246456    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:00.246960 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:00.246578    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:00.246960 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:00.246909    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:00.247108 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:00.247023    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:01.246360 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:01.246329    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:01.246558 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:01.246470    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:01.755769 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:01.755736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:01.755944 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:01.755884    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:01.755944 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:01.755942    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs podName:afc8521c-7a71-4ec0-9e3a-53d4512d5ae3 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:09.755923628 +0000 UTC m=+17.971440278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs") pod "network-metrics-daemon-6jp6n" (UID: "afc8521c-7a71-4ec0-9e3a-53d4512d5ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:01.856699 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:01.856651    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:01.857146 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:01.856900    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh48c\" (UniqueName: \"kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c\") pod \"network-check-target-kffxs\" (UID: \"a4544365-5759-41c2-8b2a-0f96b508e3d5\") " pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:01.857146 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:01.857012    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:01.857146 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:01.857045    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:01.857146 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:01.857059    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:01.857146 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:01.857071    2569 projected.go:196] Error preparing data for projected volume kube-api-access-bh48c for pod openshift-network-diagnostics/network-check-target-kffxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:01.857146 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:01.857078    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret podName:79d7959e-9636-495a-930c-a09f71fb6c57 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:09.8570599 +0000 UTC m=+18.072576551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret") pod "global-pull-secret-syncer-xdsfj" (UID: "79d7959e-9636-495a-930c-a09f71fb6c57") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:01.857146 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:01.857107    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c podName:a4544365-5759-41c2-8b2a-0f96b508e3d5 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:09.857096634 +0000 UTC m=+18.072613279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bh48c" (UniqueName: "kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c") pod "network-check-target-kffxs" (UID: "a4544365-5759-41c2-8b2a-0f96b508e3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:02.249732 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:02.249700    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:02.249732 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:02.249728    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:02.250247 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:02.249829    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:02.250247 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:02.249942    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:03.246113 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:03.246075    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:03.246299 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:03.246222    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:04.246234 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:04.246191    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:04.246234 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:04.246215    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:04.246746 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:04.246330    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:04.246746 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:04.246422    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:05.246540 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:05.246508    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:05.247016 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:05.246703    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:06.245749 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:06.245711    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:06.245749 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:06.245741    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:06.245969 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:06.245840    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:06.246039 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:06.245970    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:07.246321 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:07.246289    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:07.246771 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:07.246423    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:08.246656 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:08.246618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:08.247142 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:08.246749    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:08.247142 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:08.246806    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:08.247142 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:08.246920    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:09.245810 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:09.245778    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:09.246006 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:09.245892    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:09.817423 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:09.817384    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:09.817856 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:09.817567    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:09.817856 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:09.817645    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs podName:afc8521c-7a71-4ec0-9e3a-53d4512d5ae3 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:25.817625426 +0000 UTC m=+34.033142072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs") pod "network-metrics-daemon-6jp6n" (UID: "afc8521c-7a71-4ec0-9e3a-53d4512d5ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:09.918286 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:09.918250    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh48c\" (UniqueName: \"kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c\") pod \"network-check-target-kffxs\" (UID: \"a4544365-5759-41c2-8b2a-0f96b508e3d5\") " pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:09.918448 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:09.918302    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:09.918448 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:09.918399    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:09.918448 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:09.918415    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:09.918448 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:09.918433    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:09.918448 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:09.918447    2569 projected.go:196] Error preparing data for projected volume kube-api-access-bh48c for pod openshift-network-diagnostics/network-check-target-kffxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:09.918657 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:09.918462    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret podName:79d7959e-9636-495a-930c-a09f71fb6c57 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:25.918445425 +0000 UTC m=+34.133962072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret") pod "global-pull-secret-syncer-xdsfj" (UID: "79d7959e-9636-495a-930c-a09f71fb6c57") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:09.918657 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:09.918483    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c podName:a4544365-5759-41c2-8b2a-0f96b508e3d5 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:25.918472703 +0000 UTC m=+34.133989337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bh48c" (UniqueName: "kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c") pod "network-check-target-kffxs" (UID: "a4544365-5759-41c2-8b2a-0f96b508e3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:10.246557 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:10.246515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:10.246739 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:10.246525    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:10.246739 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:10.246643    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:10.246833 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:10.246734    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:11.245932 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:11.245894    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:11.246387 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:11.246041    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:12.246267 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:12.246236    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:12.246727 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:12.246343    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:12.246727 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:12.246408    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:12.246727 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:12.246513    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:13.245882 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:13.245855    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:13.246030 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:13.245950    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:14.246725 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.246303    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:14.247484 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.246314    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:14.247484 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:14.246829    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:14.247484 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:14.246956    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:14.365534 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.365449    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4vq25" event={"ID":"c599f721-e2df-422a-bf36-6319e85593d2","Type":"ContainerStarted","Data":"342a9c87b25c376d393a296157cb24c96b4a53190ae91f052198ad28ef9357a4"}
Jul 28 12:34:14.367152 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.367009    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e25259e-78a2-4063-8946-e80c2954e647" containerID="a824a72c367be68a09c7b62639de3479dd7bc6498b36b8dd6bee26cfa28a6dd7" exitCode=0
Jul 28 12:34:14.367282 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.367232    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn8wq" event={"ID":"0e25259e-78a2-4063-8946-e80c2954e647","Type":"ContainerDied","Data":"a824a72c367be68a09c7b62639de3479dd7bc6498b36b8dd6bee26cfa28a6dd7"}
Jul 28 12:34:14.368699 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.368674    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kr4z5" event={"ID":"c533d0e1-4c5f-4cb4-8259-b83b72a39e1d","Type":"ContainerStarted","Data":"32f9cb3e780e7fc44363e16821ad33d0c4dcb2f7d91c50d60f77d8be37f3e0a6"}
Jul 28 12:34:14.370345 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.370319    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7ng8z" event={"ID":"b9de0429-8e9d-4263-8d35-b8088687e3cc","Type":"ContainerStarted","Data":"a28e46803ac90382340ee06d62959c9507070d82759a690579ee2503ea0c7b5d"}
Jul 28 12:34:14.373420 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.373396    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf" event={"ID":"69344ace-2af3-4833-bc5f-cc6e74c00933","Type":"ContainerStarted","Data":"830aefa78d5b33d9873498e1a6e7e6f35ca31ca2c21bf8b0316e775c627fcd02"}
Jul 28 12:34:14.373547 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.373461    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf" event={"ID":"69344ace-2af3-4833-bc5f-cc6e74c00933","Type":"ContainerStarted","Data":"515893bed00ad74dbc0e01dfeca286f3081ed5eb86c289fcb143a4d9dec51693"}
Jul 28 12:34:14.373547 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.373505    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf" event={"ID":"69344ace-2af3-4833-bc5f-cc6e74c00933","Type":"ContainerStarted","Data":"f9cfacd0c886ca14907d7ec5cf352feb17a969671b33352f009f7cf9adc85c7d"}
Jul 28 12:34:14.373654 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.373549    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf" event={"ID":"69344ace-2af3-4833-bc5f-cc6e74c00933","Type":"ContainerStarted","Data":"795da22f0f5eda1987c656a883b13752700693df0902c037075e48b1c9506e4d"}
Jul 28 12:34:14.373654 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.373595    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf" event={"ID":"69344ace-2af3-4833-bc5f-cc6e74c00933","Type":"ContainerStarted","Data":"31b31b5eb9ed4ba7a1cd8a2f6f762d7b7d0a0941f2e7235e33e1f65fec658795"}
Jul 28 12:34:14.373654 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.373641    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf" event={"ID":"69344ace-2af3-4833-bc5f-cc6e74c00933","Type":"ContainerStarted","Data":"84f9493b0ac1fb6fc0f05f3c63056ce2e10def1bae78e282aa4f135bb946e80d"}
Jul 28 12:34:14.375267 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.374709    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zkstz" event={"ID":"793c18e4-9309-4da3-956e-5791d83891cf","Type":"ContainerStarted","Data":"3f4c4b4e50e4f71a23bb70b59e9db912c27c8387dc6a7794d12e2a99e4441d0b"}
Jul 28 12:34:14.376310 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.376286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lcl5q" event={"ID":"fbe4ae7f-5807-4215-8fc4-a708450f9777","Type":"ContainerStarted","Data":"781fb2da275d159c73c6190373ac411e4a3f3596a51a408ff02a39dd10091c2d"}
Jul 28 12:34:14.377840 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.377816    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r" event={"ID":"46cbac40-95c4-4172-b7d9-ffe38426e007","Type":"ContainerStarted","Data":"5c0c3ab8be9e73d1fd1643d1fb016cf9ef0903d0c68c4e37be3be8f5ad7875d3"}
Jul 28 12:34:14.384926 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.384882    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4vq25" podStartSLOduration=4.106910755 podStartE2EDuration="22.384870682s" totalImagesPullingTime="18.277959927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:14.384626463 +0000 UTC m=+22.600143113" watchObservedRunningTime="2026-07-28 12:34:14.384870682 +0000 UTC m=+22.600387366"
Jul 28 12:34:14.385507 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.385479    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" podStartSLOduration=21.385471848 podStartE2EDuration="21.385471848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:33:57.352253179 +0000 UTC m=+5.567769827" watchObservedRunningTime="2026-07-28 12:34:14.385471848 +0000 UTC m=+22.600988499"
Jul 28 12:34:14.432623 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.432570    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lcl5q" podStartSLOduration=12.2248198 podStartE2EDuration="22.432556107s" totalImagesPullingTime="10.207736307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:14.4323332 +0000 UTC m=+22.647849852" watchObservedRunningTime="2026-07-28 12:34:14.432556107 +0000 UTC m=+22.648072758"
Jul 28 12:34:14.474279 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.474224    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kr4z5" podStartSLOduration=4.164328566 podStartE2EDuration="22.47420918s" totalImagesPullingTime="18.309880614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:14.474028315 +0000 UTC m=+22.689544968" watchObservedRunningTime="2026-07-28 12:34:14.47420918 +0000 UTC m=+22.689725832"
Jul 28 12:34:14.528323 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.528278    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zkstz" podStartSLOduration=4.252254714 podStartE2EDuration="22.528263424s" totalImagesPullingTime="18.27600871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:14.500888491 +0000 UTC m=+22.716405138" watchObservedRunningTime="2026-07-28 12:34:14.528263424 +0000 UTC m=+22.743780096"
Jul 28 12:34:14.528493 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:14.528373    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7ng8z" podStartSLOduration=4.18996773 podStartE2EDuration="22.528369723s" totalImagesPullingTime="18.338401993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:14.528341978 +0000 UTC m=+22.743858640" watchObservedRunningTime="2026-07-28 12:34:14.528369723 +0000 UTC m=+22.743886375"
Jul 28 12:34:15.034568 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:15.034513    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 12:34:15.147972 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:15.147854    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:34:15.034531103Z","UUID":"74a0c402-31ca-41f2-87fa-5f8ea51fd3d9","Handler":null,"Name":"","Endpoint":""}
Jul 28 12:34:15.149844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:15.149776    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 12:34:15.149844 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:15.149824    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 12:34:15.246464 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:15.246434    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:15.246665 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:15.246565    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:15.382223 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:15.382133    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wjhph" event={"ID":"ae3368dc-9e5f-44c0-8f49-532e8336cbb7","Type":"ContainerStarted","Data":"3b185ffa75a777242b2118fefffbbcf05ec3e4b49f4d900855031f9aad99b188"}
Jul 28 12:34:15.384147 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:15.384112    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r" event={"ID":"46cbac40-95c4-4172-b7d9-ffe38426e007","Type":"ContainerStarted","Data":"283fd43626d0dbb2d3bf49f14c1b7fb66ad07d8d3833cef61308b3641c1b1461"}
Jul 28 12:34:15.401158 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:15.401104    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wjhph" podStartSLOduration=5.128676681 podStartE2EDuration="23.401085154s" totalImagesPullingTime="18.272408473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:15.400328432 +0000 UTC m=+23.615845085" watchObservedRunningTime="2026-07-28 12:34:15.401085154 +0000 UTC m=+23.616601809"
Jul 28 12:34:16.246029 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:16.246002    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:16.246164 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:16.246101    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:16.246164 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:16.246129    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:16.246283 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:16.246178    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:16.389142 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:16.389077    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r" event={"ID":"46cbac40-95c4-4172-b7d9-ffe38426e007","Type":"ContainerStarted","Data":"d379040e042bb8142dc6ecfe555b843deb44aef50b43e222f7207d5b6f986f84"}
Jul 28 12:34:16.393318 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:16.393292    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf" event={"ID":"69344ace-2af3-4833-bc5f-cc6e74c00933","Type":"ContainerStarted","Data":"f2e2f7d680e9c763eddddea5bdfeb107752eb5a354a1d53468c9a0257854e536"}
Jul 28 12:34:16.408077 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:16.408035    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqp5r" podStartSLOduration=3.932098602 podStartE2EDuration="24.408022312s" totalImagesPullingTime="20.47592371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:16.40715249 +0000 UTC m=+24.622669142" watchObservedRunningTime="2026-07-28 12:34:16.408022312 +0000 UTC m=+24.623538964"
Jul 28 12:34:17.245757 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:17.245726    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:17.246041 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:17.245860    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:17.676823 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:17.676569    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:34:17.677292 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:17.677216    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:34:18.246183 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:18.246147    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:18.246357 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:18.246153    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:18.246357 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:18.246279    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:18.246469 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:18.246367    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:18.397980 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:18.397948    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:34:18.398461 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:18.398446    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lcl5q"
Jul 28 12:34:19.245730 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:19.245563    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:19.246382 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:19.245805    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:19.401186 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:19.401081    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e25259e-78a2-4063-8946-e80c2954e647" containerID="04dd91664704f4cc70c5a18d3a63d0daeb18e426fdb1983836017fc67b48d635" exitCode=0
Jul 28 12:34:19.401186 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:19.401166    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn8wq" event={"ID":"0e25259e-78a2-4063-8946-e80c2954e647","Type":"ContainerDied","Data":"04dd91664704f4cc70c5a18d3a63d0daeb18e426fdb1983836017fc67b48d635"}
Jul 28 12:34:19.404781 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:19.404757    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf" event={"ID":"69344ace-2af3-4833-bc5f-cc6e74c00933","Type":"ContainerStarted","Data":"8bd242227a0fc1ba9f0098a84da8a0683116df3a93e1167302aa097c9af56351"}
Jul 28 12:34:19.404979 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:19.404955    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:34:19.405063 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:19.405051    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:34:19.405117 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:19.405101    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:34:19.421096 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:19.420976    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:34:19.421192 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:19.421175    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:34:19.457294 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:19.457256    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf" podStartSLOduration=9.111074672 podStartE2EDuration="27.457242733s" totalImagesPullingTime="18.346168061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:19.457114598 +0000 UTC m=+27.672631250" watchObservedRunningTime="2026-07-28 12:34:19.457242733 +0000 UTC m=+27.672759385"
Jul 28 12:34:20.246087 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:20.246053    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:20.246593 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:20.246170    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:20.246593 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:20.246199    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:20.246593 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:20.246290    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:20.773382 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:20.773300    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kffxs"]
Jul 28 12:34:20.773518 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:20.773421    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:20.773578 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:20.773534    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:20.773965 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:20.773941    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdsfj"]
Jul 28 12:34:20.774089 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:20.774039    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:20.774133 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:20.774120    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:20.775690 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:20.775670    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jp6n"]
Jul 28 12:34:20.775828 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:20.775756    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:20.775893 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:20.775833    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:21.411327 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:21.411290    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e25259e-78a2-4063-8946-e80c2954e647" containerID="81f5451dc6afe735a90572d30af4a54f27e83c68bc09a1f923a82c501833ff8b" exitCode=0
Jul 28 12:34:21.411856 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:21.411377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn8wq" event={"ID":"0e25259e-78a2-4063-8946-e80c2954e647","Type":"ContainerDied","Data":"81f5451dc6afe735a90572d30af4a54f27e83c68bc09a1f923a82c501833ff8b"}
Jul 28 12:34:22.246765 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:22.246727    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:22.246945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:22.246796    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:22.246945 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:22.246816    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:22.246945 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:22.246882    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:22.246945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:22.246904    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:22.247111 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:22.247022    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:23.418117 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:23.417918    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e25259e-78a2-4063-8946-e80c2954e647" containerID="9d4102edc73c794c3d24b6df72a3f5d7b6a528f6c2f0e5672b717ba0cda08027" exitCode=0
Jul 28 12:34:23.418117 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:23.418014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn8wq" event={"ID":"0e25259e-78a2-4063-8946-e80c2954e647","Type":"ContainerDied","Data":"9d4102edc73c794c3d24b6df72a3f5d7b6a528f6c2f0e5672b717ba0cda08027"}
Jul 28 12:34:24.246136 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:24.246102    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:24.246319 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:24.246102    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:24.246319 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:24.246208    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:24.246319 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:24.246236    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:24.246468 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:24.246315    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:24.246468 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:24.246388    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:25.834451 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:25.834417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:25.834958 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:25.834552    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:25.834958 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:25.834609    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs podName:afc8521c-7a71-4ec0-9e3a-53d4512d5ae3 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:57.834589904 +0000 UTC m=+66.050106553 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs") pod "network-metrics-daemon-6jp6n" (UID: "afc8521c-7a71-4ec0-9e3a-53d4512d5ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:25.934736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:25.934702    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh48c\" (UniqueName: \"kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c\") pod \"network-check-target-kffxs\" (UID: \"a4544365-5759-41c2-8b2a-0f96b508e3d5\") " pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:25.934736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:25.934743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:25.934959 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:25.934866    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:25.934959 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:25.934915    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret podName:79d7959e-9636-495a-930c-a09f71fb6c57 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:57.934902145 +0000 UTC m=+66.150418788 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret") pod "global-pull-secret-syncer-xdsfj" (UID: "79d7959e-9636-495a-930c-a09f71fb6c57") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:25.934959 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:25.934869    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:25.934959 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:25.934942    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:25.934959 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:25.934954    2569 projected.go:196] Error preparing data for projected volume kube-api-access-bh48c for pod openshift-network-diagnostics/network-check-target-kffxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:25.935244 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:25.935008    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c podName:a4544365-5759-41c2-8b2a-0f96b508e3d5 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:57.934981077 +0000 UTC m=+66.150497724 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bh48c" (UniqueName: "kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c") pod "network-check-target-kffxs" (UID: "a4544365-5759-41c2-8b2a-0f96b508e3d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:26.245951 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:26.245918    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:26.246143 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:26.245918    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:26.246143 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:26.245933    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:26.246143 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:26.246112    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:26.246349 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:26.246162    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:26.246349 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:26.246209    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:28.246042 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:28.246007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:28.246640 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:28.246007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:28.246640 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:28.246129    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:28.246640 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:28.246015    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:28.246640 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:28.246228    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:28.246640 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:28.246351    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:29.433134 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:29.433105    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e25259e-78a2-4063-8946-e80c2954e647" containerID="696049e38ebaf4e9300bafe8daf2c717cd905b40edea828bf4e1b6b47e273592" exitCode=0
Jul 28 12:34:29.433570 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:29.433167    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn8wq" event={"ID":"0e25259e-78a2-4063-8946-e80c2954e647","Type":"ContainerDied","Data":"696049e38ebaf4e9300bafe8daf2c717cd905b40edea828bf4e1b6b47e273592"}
Jul 28 12:34:30.245922 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:30.245888    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:30.246148 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:30.245888    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:30.246148 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:30.245888    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:30.246148 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:30.246024    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:30.246148 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:30.246061    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:30.246148 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:30.246117    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:30.438272 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:30.438239    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e25259e-78a2-4063-8946-e80c2954e647" containerID="d84217bd3b1c031da9419b38a370ced1cca0a05cb8043ef8d9291d871fef8e8e" exitCode=0
Jul 28 12:34:30.438608 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:30.438290    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn8wq" event={"ID":"0e25259e-78a2-4063-8946-e80c2954e647","Type":"ContainerDied","Data":"d84217bd3b1c031da9419b38a370ced1cca0a05cb8043ef8d9291d871fef8e8e"}
Jul 28 12:34:31.443846 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:31.443811    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn8wq" event={"ID":"0e25259e-78a2-4063-8946-e80c2954e647","Type":"ContainerStarted","Data":"b4fab9553d17ba1762b8401c96f43a2b006ca4cb17f33a857b75c1f01bd11021"}
Jul 28 12:34:31.470221 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:31.470165    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pn8wq" podStartSLOduration=9.167151068 podStartE2EDuration="39.470147036s" totalImagesPullingTime="30.302995968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:31.469843733 +0000 UTC m=+39.685360386" watchObservedRunningTime="2026-07-28 12:34:31.470147036 +0000 UTC m=+39.685663688"
Jul 28 12:34:32.247083 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:32.247053    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:32.247266 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:32.247139    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:32.247266 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:32.247242    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:32.247384 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:32.247345    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:32.247435 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:32.247402    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:32.247503 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:32.247485    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:34.246289 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:34.246254    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:34.246289 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:34.246291    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:34.246755 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:34.246258    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:34.246755 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:34.246359    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:34.246755 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:34.246475    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:34.246755 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:34.246621    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:36.245817 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:36.245778    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:36.245817 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:36.245809    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:36.246472 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:36.245778    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:36.246472 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:36.245884    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:36.246472 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:36.245968    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:36.246472 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:36.246055    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:38.246026 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:38.245973    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:38.246516 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:38.245977    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:38.246516 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:38.246117    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:38.246516 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:38.246125    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:38.246516 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:38.246233    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:38.246516 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:38.246311    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:40.245629 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:40.245601    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:40.246014 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:40.245633    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:40.246014 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:40.245609    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:40.246014 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:40.245716    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:40.246014 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:40.245789    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:40.246014 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:40.245845    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:41.246306 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:41.246269    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:41.246741 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:41.246388    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:42.247296 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:42.247256    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:42.247766 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:42.247337    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:42.247766 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:42.247342    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:42.247766 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:42.247440    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:43.246114 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:43.246080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:43.246316 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:43.246199    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:44.245782 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:44.245751    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:44.246170 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:44.245851    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kffxs" podUID="a4544365-5759-41c2-8b2a-0f96b508e3d5"
Jul 28 12:34:44.246170 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:44.245926    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:44.246170 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:44.246032    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jp6n" podUID="afc8521c-7a71-4ec0-9e3a-53d4512d5ae3"
Jul 28 12:34:45.246043 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:45.246013    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:45.246405 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:34:45.246113    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdsfj" podUID="79d7959e-9636-495a-930c-a09f71fb6c57"
Jul 28 12:34:46.127203 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.127173    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeReady"
Jul 28 12:34:46.127366 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.127295    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 12:34:46.172707 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.172677    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dc9fbdd49-tlf2w"]
Jul 28 12:34:46.189712 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.189637    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.190469 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.190443    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dc9fbdd49-tlf2w"]
Jul 28 12:34:46.193335 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.193317    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpxsc\""
Jul 28 12:34:46.193747 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.193729    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 12:34:46.193834 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.193729    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 12:34:46.195579 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.195557    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 12:34:46.199209 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.199192    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 12:34:46.222475 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.222456    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hwfbj"]
Jul 28 12:34:46.244502 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.244453    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f58jt"]
Jul 28 12:34:46.244581 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.244569    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.247615 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.247600    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g8tcp\""
Jul 28 12:34:46.247905 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.247714    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 12:34:46.247980 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.247932    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 12:34:46.264714 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.264687    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.267860 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.267842    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hwfbj"]
Jul 28 12:34:46.267998 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.267971    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:46.268063 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.268005    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:46.269146 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.269124    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f58jt"]
Jul 28 12:34:46.273222 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.273203    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 12:34:46.273341 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.273325    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:34:46.273406 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.273393    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:34:46.273817 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.273560    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 12:34:46.273817 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.273647    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:34:46.273817 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.273680    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jpxcx\""
Jul 28 12:34:46.273817 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.273756    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 12:34:46.274337 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.274317    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 12:34:46.274436 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.274353    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7n8fl\""
Jul 28 12:34:46.274436 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.274415    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nnkkf\""
Jul 28 12:34:46.289817 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.289796    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f33c0ba-7ac0-4906-823e-8632e6008279-image-registry-private-configuration\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.289907 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.289823    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560ade9a-e010-41ab-9174-54a41e8d323e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.289907 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.289842    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/560ade9a-e010-41ab-9174-54a41e8d323e-crio-socket\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.289907 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.289857    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560ade9a-e010-41ab-9174-54a41e8d323e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.289907 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.289872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqfzg\" (UniqueName: \"kubernetes.io/projected/13ffc097-2363-4b74-a9d8-f6865e6f22dc-kube-api-access-mqfzg\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.290125 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.289936    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbsr7\" (UniqueName: \"kubernetes.io/projected/8f33c0ba-7ac0-4906-823e-8632e6008279-kube-api-access-dbsr7\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.290125 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290000    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f33c0ba-7ac0-4906-823e-8632e6008279-registry-certificates\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.290125 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290040    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f33c0ba-7ac0-4906-823e-8632e6008279-registry-tls\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.290125 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290072    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13ffc097-2363-4b74-a9d8-f6865e6f22dc-config-volume\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.290125 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290103    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f33c0ba-7ac0-4906-823e-8632e6008279-installation-pull-secrets\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.290288 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxsmn\" (UniqueName: \"kubernetes.io/projected/560ade9a-e010-41ab-9174-54a41e8d323e-kube-api-access-nxsmn\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.290288 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f33c0ba-7ac0-4906-823e-8632e6008279-trusted-ca\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.290288 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290196    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f33c0ba-7ac0-4906-823e-8632e6008279-bound-sa-token\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.290288 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290212    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13ffc097-2363-4b74-a9d8-f6865e6f22dc-metrics-tls\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.290288 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f33c0ba-7ac0-4906-823e-8632e6008279-ca-trust-extracted\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.290288 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ffc097-2363-4b74-a9d8-f6865e6f22dc-tmp-dir\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.290469 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.290304    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/560ade9a-e010-41ab-9174-54a41e8d323e-data-volume\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.333179 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.333151    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8bmdm"]
Jul 28 12:34:46.356559 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.356534    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8bmdm"
Jul 28 12:34:46.356559 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.356553    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8bmdm"]
Jul 28 12:34:46.359663 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.359641    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 12:34:46.359777 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.359714    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 12:34:46.359856 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.359842    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w8fbz\""
Jul 28 12:34:46.359896 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.359862    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 12:34:46.391216 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391192    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbsr7\" (UniqueName: \"kubernetes.io/projected/8f33c0ba-7ac0-4906-823e-8632e6008279-kube-api-access-dbsr7\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.391305 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f33c0ba-7ac0-4906-823e-8632e6008279-registry-certificates\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.391305 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f33c0ba-7ac0-4906-823e-8632e6008279-registry-tls\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.391305 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391257    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62ad7a92-64da-4091-8293-399480862c3e-cert\") pod \"ingress-canary-8bmdm\" (UID: \"62ad7a92-64da-4091-8293-399480862c3e\") " pod="openshift-ingress-canary/ingress-canary-8bmdm"
Jul 28 12:34:46.391305 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391280    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13ffc097-2363-4b74-a9d8-f6865e6f22dc-config-volume\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.391496 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391309    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f33c0ba-7ac0-4906-823e-8632e6008279-installation-pull-secrets\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.391496 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxsmn\" (UniqueName: \"kubernetes.io/projected/560ade9a-e010-41ab-9174-54a41e8d323e-kube-api-access-nxsmn\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.391496 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391364    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f33c0ba-7ac0-4906-823e-8632e6008279-trusted-ca\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.391496 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391385    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f33c0ba-7ac0-4906-823e-8632e6008279-bound-sa-token\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.391496 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391421    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13ffc097-2363-4b74-a9d8-f6865e6f22dc-metrics-tls\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.391496 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f33c0ba-7ac0-4906-823e-8632e6008279-ca-trust-extracted\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.391496 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391488    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ffc097-2363-4b74-a9d8-f6865e6f22dc-tmp-dir\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.391761 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvghz\" (UniqueName: \"kubernetes.io/projected/62ad7a92-64da-4091-8293-399480862c3e-kube-api-access-fvghz\") pod \"ingress-canary-8bmdm\" (UID: \"62ad7a92-64da-4091-8293-399480862c3e\") " pod="openshift-ingress-canary/ingress-canary-8bmdm"
Jul 28 12:34:46.391761 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391557    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/560ade9a-e010-41ab-9174-54a41e8d323e-data-volume\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.391761 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f33c0ba-7ac0-4906-823e-8632e6008279-image-registry-private-configuration\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.391761 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391611    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560ade9a-e010-41ab-9174-54a41e8d323e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.391761 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391629    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/560ade9a-e010-41ab-9174-54a41e8d323e-crio-socket\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.391761 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560ade9a-e010-41ab-9174-54a41e8d323e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.391761 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.391661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqfzg\" (UniqueName: \"kubernetes.io/projected/13ffc097-2363-4b74-a9d8-f6865e6f22dc-kube-api-access-mqfzg\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.392196 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.392176    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/560ade9a-e010-41ab-9174-54a41e8d323e-data-volume\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.392318 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.392253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ffc097-2363-4b74-a9d8-f6865e6f22dc-tmp-dir\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.392474 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.392457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/560ade9a-e010-41ab-9174-54a41e8d323e-crio-socket\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.392565 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.392544    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f33c0ba-7ac0-4906-823e-8632e6008279-registry-certificates\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.392628 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.392523    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f33c0ba-7ac0-4906-823e-8632e6008279-ca-trust-extracted\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.392778 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.392748    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f33c0ba-7ac0-4906-823e-8632e6008279-trusted-ca\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.393225 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.393201    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13ffc097-2363-4b74-a9d8-f6865e6f22dc-config-volume\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.393225 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.393222    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560ade9a-e010-41ab-9174-54a41e8d323e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.395489 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.395462    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560ade9a-e010-41ab-9174-54a41e8d323e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.395571 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.395469    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f33c0ba-7ac0-4906-823e-8632e6008279-installation-pull-secrets\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.395571 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.395516    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f33c0ba-7ac0-4906-823e-8632e6008279-image-registry-private-configuration\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.395646 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.395570    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13ffc097-2363-4b74-a9d8-f6865e6f22dc-metrics-tls\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.395678 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.395668    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f33c0ba-7ac0-4906-823e-8632e6008279-registry-tls\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.399548 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.399524    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxsmn\" (UniqueName: \"kubernetes.io/projected/560ade9a-e010-41ab-9174-54a41e8d323e-kube-api-access-nxsmn\") pod \"insights-runtime-extractor-f58jt\" (UID: \"560ade9a-e010-41ab-9174-54a41e8d323e\") " pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.399680 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.399664    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbsr7\" (UniqueName: \"kubernetes.io/projected/8f33c0ba-7ac0-4906-823e-8632e6008279-kube-api-access-dbsr7\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.400089 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.400071    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f33c0ba-7ac0-4906-823e-8632e6008279-bound-sa-token\") pod \"image-registry-dc9fbdd49-tlf2w\" (UID: \"8f33c0ba-7ac0-4906-823e-8632e6008279\") " pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.400517 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.400497    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqfzg\" (UniqueName: \"kubernetes.io/projected/13ffc097-2363-4b74-a9d8-f6865e6f22dc-kube-api-access-mqfzg\") pod \"dns-default-hwfbj\" (UID: \"13ffc097-2363-4b74-a9d8-f6865e6f22dc\") " pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.492816 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.492790    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvghz\" (UniqueName: \"kubernetes.io/projected/62ad7a92-64da-4091-8293-399480862c3e-kube-api-access-fvghz\") pod \"ingress-canary-8bmdm\" (UID: \"62ad7a92-64da-4091-8293-399480862c3e\") " pod="openshift-ingress-canary/ingress-canary-8bmdm"
Jul 28 12:34:46.492964 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.492859    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62ad7a92-64da-4091-8293-399480862c3e-cert\") pod \"ingress-canary-8bmdm\" (UID: \"62ad7a92-64da-4091-8293-399480862c3e\") " pod="openshift-ingress-canary/ingress-canary-8bmdm"
Jul 28 12:34:46.495178 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.495134    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62ad7a92-64da-4091-8293-399480862c3e-cert\") pod \"ingress-canary-8bmdm\" (UID: \"62ad7a92-64da-4091-8293-399480862c3e\") " pod="openshift-ingress-canary/ingress-canary-8bmdm"
Jul 28 12:34:46.498914 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.498898    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:46.503411 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.503394    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvghz\" (UniqueName: \"kubernetes.io/projected/62ad7a92-64da-4091-8293-399480862c3e-kube-api-access-fvghz\") pod \"ingress-canary-8bmdm\" (UID: \"62ad7a92-64da-4091-8293-399480862c3e\") " pod="openshift-ingress-canary/ingress-canary-8bmdm"
Jul 28 12:34:46.552445 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.552408    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:46.576316 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.576279    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f58jt"
Jul 28 12:34:46.653099 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.653004    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dc9fbdd49-tlf2w"]
Jul 28 12:34:46.664819 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.664580    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8bmdm"
Jul 28 12:34:46.672488 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:34:46.672459    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f33c0ba_7ac0_4906_823e_8632e6008279.slice/crio-b5576a3d6b36bb0c1da8310a2af6140c70a84c083d8f687bfb57b3d7871e9532 WatchSource:0}: Error finding container b5576a3d6b36bb0c1da8310a2af6140c70a84c083d8f687bfb57b3d7871e9532: Status 404 returned error can't find the container with id b5576a3d6b36bb0c1da8310a2af6140c70a84c083d8f687bfb57b3d7871e9532
Jul 28 12:34:46.701764 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.701736    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hwfbj"]
Jul 28 12:34:46.709136 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:34:46.709102    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ffc097_2363_4b74_a9d8_f6865e6f22dc.slice/crio-7916070c1c2f26f3c2b967056cec1428c8fd37f33a6226da18637fdb01b62ee7 WatchSource:0}: Error finding container 7916070c1c2f26f3c2b967056cec1428c8fd37f33a6226da18637fdb01b62ee7: Status 404 returned error can't find the container with id 7916070c1c2f26f3c2b967056cec1428c8fd37f33a6226da18637fdb01b62ee7
Jul 28 12:34:46.738885 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.738860    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f58jt"]
Jul 28 12:34:46.741680 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:34:46.741645    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560ade9a_e010_41ab_9174_54a41e8d323e.slice/crio-76ed4b04ce430152a06240897c310b547777b1869e3712149902210d69ddf8d4 WatchSource:0}: Error finding container 76ed4b04ce430152a06240897c310b547777b1869e3712149902210d69ddf8d4: Status 404 returned error can't find the container with id 76ed4b04ce430152a06240897c310b547777b1869e3712149902210d69ddf8d4
Jul 28 12:34:46.798724 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:46.798696    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8bmdm"]
Jul 28 12:34:46.801818 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:34:46.801795    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ad7a92_64da_4091_8293_399480862c3e.slice/crio-16e17d6885f74f9534546839fb6e3aed2d2d39d54b80aa7e2044bada5c4c701a WatchSource:0}: Error finding container 16e17d6885f74f9534546839fb6e3aed2d2d39d54b80aa7e2044bada5c4c701a: Status 404 returned error can't find the container with id 16e17d6885f74f9534546839fb6e3aed2d2d39d54b80aa7e2044bada5c4c701a
Jul 28 12:34:47.247164 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:47.246313    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:47.249807 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:47.249293    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:34:47.479676 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:47.479595    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8bmdm" event={"ID":"62ad7a92-64da-4091-8293-399480862c3e","Type":"ContainerStarted","Data":"16e17d6885f74f9534546839fb6e3aed2d2d39d54b80aa7e2044bada5c4c701a"}
Jul 28 12:34:47.483756 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:47.483625    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f58jt" event={"ID":"560ade9a-e010-41ab-9174-54a41e8d323e","Type":"ContainerStarted","Data":"76ed4b04ce430152a06240897c310b547777b1869e3712149902210d69ddf8d4"}
Jul 28 12:34:47.486293 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:47.486191    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hwfbj" event={"ID":"13ffc097-2363-4b74-a9d8-f6865e6f22dc","Type":"ContainerStarted","Data":"7916070c1c2f26f3c2b967056cec1428c8fd37f33a6226da18637fdb01b62ee7"}
Jul 28 12:34:47.491962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:47.491130    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w" event={"ID":"8f33c0ba-7ac0-4906-823e-8632e6008279","Type":"ContainerStarted","Data":"193793b1430f7e981fc047c2af4b80acbeeb3287baec3de676c3d7a6a0d12c9c"}
Jul 28 12:34:47.491962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:47.491168    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w" event={"ID":"8f33c0ba-7ac0-4906-823e-8632e6008279","Type":"ContainerStarted","Data":"b5576a3d6b36bb0c1da8310a2af6140c70a84c083d8f687bfb57b3d7871e9532"}
Jul 28 12:34:47.491962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:47.491897    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:34:47.513779 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:47.513676    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w" podStartSLOduration=2.51365809 podStartE2EDuration="2.51365809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:47.512568551 +0000 UTC m=+55.728085204" watchObservedRunningTime="2026-07-28 12:34:47.51365809 +0000 UTC m=+55.729174723"
Jul 28 12:34:49.499610 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:49.499574    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hwfbj" event={"ID":"13ffc097-2363-4b74-a9d8-f6865e6f22dc","Type":"ContainerStarted","Data":"59c6f67cfd66a3290cf245b45d25163bf91be5778fbdcf5153ceaa1b4ecf8cc8"}
Jul 28 12:34:49.500048 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:49.499678    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hwfbj" event={"ID":"13ffc097-2363-4b74-a9d8-f6865e6f22dc","Type":"ContainerStarted","Data":"bfa0601e67b4e467461fc27cf11ba07efbdcaef1cadbf9b9ff46d436fbbc52ef"}
Jul 28 12:34:49.519033 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:49.518961    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hwfbj" podStartSLOduration=1.61492316 podStartE2EDuration="3.518944307s" totalImagesPullingTime="1.904021147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:49.518048214 +0000 UTC m=+57.733564868" watchObservedRunningTime="2026-07-28 12:34:49.518944307 +0000 UTC m=+57.734460958"
Jul 28 12:34:50.502326 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:50.502292    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:34:51.421307 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:51.421279    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-45gqf"
Jul 28 12:34:52.509041 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:52.509009    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8bmdm" event={"ID":"62ad7a92-64da-4091-8293-399480862c3e","Type":"ContainerStarted","Data":"d5f9c2dd00671e23c1f2da3cd96c3746f2e8511f15acbf907c2e137fe4d1e2c7"}
Jul 28 12:34:52.526577 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:52.526530    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8bmdm" podStartSLOduration=0.91943377 podStartE2EDuration="6.526515199s" totalImagesPullingTime="5.607081429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:52.526175252 +0000 UTC m=+60.741691926" watchObservedRunningTime="2026-07-28 12:34:52.526515199 +0000 UTC m=+60.742031850"
Jul 28 12:34:53.514459 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:53.514417    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f58jt" event={"ID":"560ade9a-e010-41ab-9174-54a41e8d323e","Type":"ContainerStarted","Data":"5bf12cd3ed3f4ce50196d6bd3a6ea1e98f4b561ec2e3114c07cc804c98d09b6a"}
Jul 28 12:34:54.039594 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.039556    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-q8qms"]
Jul 28 12:34:54.041713 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.041687    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.044941 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.044913    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 12:34:54.046033 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.045971    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 12:34:54.046156 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.046098    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 12:34:54.046156 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.046114    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h64mr\""
Jul 28 12:34:54.046318 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.046223    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 12:34:54.046318 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.046234    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 12:34:54.053792 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.053739    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-q8qms"]
Jul 28 12:34:54.154577 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.154536    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbv57\" (UniqueName: \"kubernetes.io/projected/42193b4e-1fdd-45fb-9e27-91533f3b564f-kube-api-access-xbv57\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.154766 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.154630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42193b4e-1fdd-45fb-9e27-91533f3b564f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.154766 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.154752    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42193b4e-1fdd-45fb-9e27-91533f3b564f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.154929 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.154907    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42193b4e-1fdd-45fb-9e27-91533f3b564f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.255474 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.255435    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42193b4e-1fdd-45fb-9e27-91533f3b564f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.255638 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.255481    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbv57\" (UniqueName: \"kubernetes.io/projected/42193b4e-1fdd-45fb-9e27-91533f3b564f-kube-api-access-xbv57\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.255638 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.255548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42193b4e-1fdd-45fb-9e27-91533f3b564f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.255638 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.255634    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42193b4e-1fdd-45fb-9e27-91533f3b564f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.256391 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.256367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42193b4e-1fdd-45fb-9e27-91533f3b564f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.258351 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.258324    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42193b4e-1fdd-45fb-9e27-91533f3b564f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.258465 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.258386    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42193b4e-1fdd-45fb-9e27-91533f3b564f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.266203 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.266177    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbv57\" (UniqueName: \"kubernetes.io/projected/42193b4e-1fdd-45fb-9e27-91533f3b564f-kube-api-access-xbv57\") pod \"prometheus-operator-84d69f794d-q8qms\" (UID: \"42193b4e-1fdd-45fb-9e27-91533f3b564f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:54.352055 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:54.351954    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms"
Jul 28 12:34:56.041466 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.041429    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69bbdccd5c-ltgjb"]
Jul 28 12:34:56.049758 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.049729    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.053405 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.053381    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:34:56.053405 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.053399    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 12:34:56.053596 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.053433    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-224r5\""
Jul 28 12:34:56.053596 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.053404    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:34:56.053596 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.053455    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 12:34:56.053596 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.053383    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 12:34:56.053596 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.053434    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 12:34:56.053596 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.053383    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 12:34:56.055635 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.055615    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69bbdccd5c-ltgjb"]
Jul 28 12:34:56.174204 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.174165    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-serving-cert\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.174379 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.174221    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-config\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.174379 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.174352    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khvvz\" (UniqueName: \"kubernetes.io/projected/77fac7f3-5fa0-4364-afd1-c8739b27d163-kube-api-access-khvvz\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.174483 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.174387    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-oauth-config\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.174483 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.174470    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-service-ca\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.174599 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.174582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-oauth-serving-cert\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.275450 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.275418    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-service-ca\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.275683 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.275486    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-oauth-serving-cert\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.275683 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.275514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-serving-cert\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.275683 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.275546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-config\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.275683 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.275599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khvvz\" (UniqueName: \"kubernetes.io/projected/77fac7f3-5fa0-4364-afd1-c8739b27d163-kube-api-access-khvvz\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.275683 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.275620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-oauth-config\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.276313 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.276290    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-service-ca\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.276418 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.276395    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-oauth-serving-cert\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.276684 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.276666    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-config\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.277846 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.277825    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-oauth-config\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.278372 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.278353    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-serving-cert\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.285135 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.285114    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khvvz\" (UniqueName: \"kubernetes.io/projected/77fac7f3-5fa0-4364-afd1-c8739b27d163-kube-api-access-khvvz\") pod \"console-69bbdccd5c-ltgjb\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") " pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.315530 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.315505    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-q8qms"]
Jul 28 12:34:56.318067 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:34:56.318044    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42193b4e_1fdd_45fb_9e27_91533f3b564f.slice/crio-07e54aa99789a6ac5b2ac2cf13df93fdcc17f79b7636c830babb998322f20a70 WatchSource:0}: Error finding container 07e54aa99789a6ac5b2ac2cf13df93fdcc17f79b7636c830babb998322f20a70: Status 404 returned error can't find the container with id 07e54aa99789a6ac5b2ac2cf13df93fdcc17f79b7636c830babb998322f20a70
Jul 28 12:34:56.360772 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.360748    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:34:56.477875 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.477843    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69bbdccd5c-ltgjb"]
Jul 28 12:34:56.491857 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:34:56.491831    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fac7f3_5fa0_4364_afd1_c8739b27d163.slice/crio-145b9af5871f35ce5bebd96e892b09d2e67e8b22eb997a516f12dd41da5a2adf WatchSource:0}: Error finding container 145b9af5871f35ce5bebd96e892b09d2e67e8b22eb997a516f12dd41da5a2adf: Status 404 returned error can't find the container with id 145b9af5871f35ce5bebd96e892b09d2e67e8b22eb997a516f12dd41da5a2adf
Jul 28 12:34:56.524530 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.524501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms" event={"ID":"42193b4e-1fdd-45fb-9e27-91533f3b564f","Type":"ContainerStarted","Data":"07e54aa99789a6ac5b2ac2cf13df93fdcc17f79b7636c830babb998322f20a70"}
Jul 28 12:34:56.525580 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.525547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69bbdccd5c-ltgjb" event={"ID":"77fac7f3-5fa0-4364-afd1-c8739b27d163","Type":"ContainerStarted","Data":"145b9af5871f35ce5bebd96e892b09d2e67e8b22eb997a516f12dd41da5a2adf"}
Jul 28 12:34:56.527023 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:56.527003    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f58jt" event={"ID":"560ade9a-e010-41ab-9174-54a41e8d323e","Type":"ContainerStarted","Data":"c231e5bbe201326c31f342624db05846e245a97545520d8f65a7404bf62eaab0"}
Jul 28 12:34:57.889724 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:57.889682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:57.892375 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:57.892349    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:34:57.903189 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:57.903158    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc8521c-7a71-4ec0-9e3a-53d4512d5ae3-metrics-certs\") pod \"network-metrics-daemon-6jp6n\" (UID: \"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3\") " pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:57.982643 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:57.982607    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7n8fl\""
Jul 28 12:34:57.991029 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:57.990982    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jp6n"
Jul 28 12:34:57.991165 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:57.991023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh48c\" (UniqueName: \"kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c\") pod \"network-check-target-kffxs\" (UID: \"a4544365-5759-41c2-8b2a-0f96b508e3d5\") " pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:57.991165 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:57.991073    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:57.993544 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:57.993525    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:34:57.993934 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:57.993914    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:34:58.003885 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.003863    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:34:58.005409 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.005137    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79d7959e-9636-495a-930c-a09f71fb6c57-original-pull-secret\") pod \"global-pull-secret-syncer-xdsfj\" (UID: \"79d7959e-9636-495a-930c-a09f71fb6c57\") " pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:58.015075 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.015031    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh48c\" (UniqueName: \"kubernetes.io/projected/a4544365-5759-41c2-8b2a-0f96b508e3d5-kube-api-access-bh48c\") pod \"network-check-target-kffxs\" (UID: \"a4544365-5759-41c2-8b2a-0f96b508e3d5\") " pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:58.060575 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.060458    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdsfj"
Jul 28 12:34:58.204150 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.204063    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jp6n"]
Jul 28 12:34:58.258424 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.258397    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdsfj"]
Jul 28 12:34:58.293147 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.289525    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jpxcx\""
Jul 28 12:34:58.296256 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.296038    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:34:58.460811 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.460708    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kffxs"]
Jul 28 12:34:58.464513 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:34:58.464484    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4544365_5759_41c2_8b2a_0f96b508e3d5.slice/crio-165efe3d47b67b2396d2c8d4a290938b9381d9a4af9b3bdf88304624146dc363 WatchSource:0}: Error finding container 165efe3d47b67b2396d2c8d4a290938b9381d9a4af9b3bdf88304624146dc363: Status 404 returned error can't find the container with id 165efe3d47b67b2396d2c8d4a290938b9381d9a4af9b3bdf88304624146dc363
Jul 28 12:34:58.535907 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.535873    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kffxs" event={"ID":"a4544365-5759-41c2-8b2a-0f96b508e3d5","Type":"ContainerStarted","Data":"165efe3d47b67b2396d2c8d4a290938b9381d9a4af9b3bdf88304624146dc363"}
Jul 28 12:34:58.537657 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.537588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jp6n" event={"ID":"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3","Type":"ContainerStarted","Data":"2dfb3d53e73b8dcee002e7762deef04114f954999eef018756c4c490bd535f1c"}
Jul 28 12:34:58.539267 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.539237    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdsfj" event={"ID":"79d7959e-9636-495a-930c-a09f71fb6c57","Type":"ContainerStarted","Data":"541d41b248655fc826dec8f05297c7ce5e10adb7eda60c40dc945703c48003fc"}
Jul 28 12:34:58.541348 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.541258    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms" event={"ID":"42193b4e-1fdd-45fb-9e27-91533f3b564f","Type":"ContainerStarted","Data":"0875e037536d29741da1c5465bbad16d9cbd7f4a3cc84761decfe5b4eab56ab3"}
Jul 28 12:34:58.541348 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.541305    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms" event={"ID":"42193b4e-1fdd-45fb-9e27-91533f3b564f","Type":"ContainerStarted","Data":"91ff96a0d3a0502538a53df1885324dbe2f4d6e0a62b69f666fa5c9d1f08fc64"}
Jul 28 12:34:58.567014 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:34:58.566841    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-q8qms" podStartSLOduration=2.790394921 podStartE2EDuration="4.56682163s" totalImagesPullingTime="1.776426709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:58.566256766 +0000 UTC m=+66.781773435" watchObservedRunningTime="2026-07-28 12:34:58.56682163 +0000 UTC m=+66.782338285"
Jul 28 12:35:00.365420 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.365381    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"]
Jul 28 12:35:00.446519 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.446478    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"]
Jul 28 12:35:00.446689 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.446612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.449650 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.449623    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 12:35:00.449977 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.449960    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 12:35:00.450366 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.450343    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7l465\""
Jul 28 12:35:00.458861 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.458839    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"]
Jul 28 12:35:00.458963 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.458881    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dbx9v"]
Jul 28 12:35:00.458963 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.458951    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.461523 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.461500    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 12:35:00.461672 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.461652    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9h8zw\""
Jul 28 12:35:00.461672 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.461656    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 12:35:00.461848 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.461694    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 12:35:00.461942 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.461927    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"]
Jul 28 12:35:00.462071 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.462056    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.464249 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.464228    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 12:35:00.465691 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.465450    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2zvmv\""
Jul 28 12:35:00.465691 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.465500    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 12:35:00.465691 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.465452    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 12:35:00.506794 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.506772    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hwfbj"
Jul 28 12:35:00.513445 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513420    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/845b2466-b87c-4d92-97ba-0f2309e87624-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.513554 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513455    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkb2\" (UniqueName: \"kubernetes.io/projected/845b2466-b87c-4d92-97ba-0f2309e87624-kube-api-access-nkkb2\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.513554 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbfnc\" (UniqueName: \"kubernetes.io/projected/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-kube-api-access-mbfnc\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.513554 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.513673 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-tls\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.513673 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513609    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-textfile\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.513673 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.513800 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513701    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/845b2466-b87c-4d92-97ba-0f2309e87624-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.513800 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513736    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/845b2466-b87c-4d92-97ba-0f2309e87624-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.513800 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513765    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.513909 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/845b2466-b87c-4d92-97ba-0f2309e87624-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.513909 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513828    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.513909 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/845b2466-b87c-4d92-97ba-0f2309e87624-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.513909 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/846183c6-9edf-4a5e-b6ee-de3752e60c06-metrics-client-ca\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.514089 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513933    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846183c6-9edf-4a5e-b6ee-de3752e60c06-sys\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.514089 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513955    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-wtmp\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.514089 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.513979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdtg\" (UniqueName: \"kubernetes.io/projected/846183c6-9edf-4a5e-b6ee-de3752e60c06-kube-api-access-8bdtg\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.514089 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.514061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/846183c6-9edf-4a5e-b6ee-de3752e60c06-root\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.514249 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.514101    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.614631 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkb2\" (UniqueName: \"kubernetes.io/projected/845b2466-b87c-4d92-97ba-0f2309e87624-kube-api-access-nkkb2\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.614631 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614633    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbfnc\" (UniqueName: \"kubernetes.io/projected/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-kube-api-access-mbfnc\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.614858 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.614858 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614706    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-tls\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.614858 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-textfile\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.614858 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614760    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.614858 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614797    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/845b2466-b87c-4d92-97ba-0f2309e87624-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.614858 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/845b2466-b87c-4d92-97ba-0f2309e87624-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.615173 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614861    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.615173 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614894    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/845b2466-b87c-4d92-97ba-0f2309e87624-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.615173 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.614923    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.615682    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.615757    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/845b2466-b87c-4d92-97ba-0f2309e87624-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.615787    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/846183c6-9edf-4a5e-b6ee-de3752e60c06-metrics-client-ca\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.615811    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.615819    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846183c6-9edf-4a5e-b6ee-de3752e60c06-sys\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.615877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-wtmp\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.615906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdtg\" (UniqueName: \"kubernetes.io/projected/846183c6-9edf-4a5e-b6ee-de3752e60c06-kube-api-access-8bdtg\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.615957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/846183c6-9edf-4a5e-b6ee-de3752e60c06-root\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.616021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.616053    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/845b2466-b87c-4d92-97ba-0f2309e87624-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.616223    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/845b2466-b87c-4d92-97ba-0f2309e87624-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.616287    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-textfile\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.616296    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/845b2466-b87c-4d92-97ba-0f2309e87624-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.616345    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/845b2466-b87c-4d92-97ba-0f2309e87624-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.616354    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/846183c6-9edf-4a5e-b6ee-de3752e60c06-root\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.616842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.616503    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/846183c6-9edf-4a5e-b6ee-de3752e60c06-sys\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.617731 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.616635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-wtmp\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.617731 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.616745    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/846183c6-9edf-4a5e-b6ee-de3752e60c06-metrics-client-ca\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.618424 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.618394    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/845b2466-b87c-4d92-97ba-0f2309e87624-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.618956 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.618911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/845b2466-b87c-4d92-97ba-0f2309e87624-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.619456 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.619430    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.619630 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.619587    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.621316 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.621293    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/846183c6-9edf-4a5e-b6ee-de3752e60c06-node-exporter-tls\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.621425 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.621393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.626766 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.626711    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbfnc\" (UniqueName: \"kubernetes.io/projected/dc80c0af-bbf0-43f8-a9fc-f72af0510f41-kube-api-access-mbfnc\") pod \"openshift-state-metrics-b8f49f5c6-j6bx6\" (UID: \"dc80c0af-bbf0-43f8-a9fc-f72af0510f41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.626910 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.626890    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkb2\" (UniqueName: \"kubernetes.io/projected/845b2466-b87c-4d92-97ba-0f2309e87624-kube-api-access-nkkb2\") pod \"kube-state-metrics-56d9f944b4-g5dt2\" (UID: \"845b2466-b87c-4d92-97ba-0f2309e87624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.628667 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.628646    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdtg\" (UniqueName: \"kubernetes.io/projected/846183c6-9edf-4a5e-b6ee-de3752e60c06-kube-api-access-8bdtg\") pod \"node-exporter-dbx9v\" (UID: \"846183c6-9edf-4a5e-b6ee-de3752e60c06\") " pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:00.756882 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.756846    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"
Jul 28 12:35:00.771234 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.771212    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"
Jul 28 12:35:00.777803 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:00.777774    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dbx9v"
Jul 28 12:35:01.431931 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.431895    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:35:01.445063 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.445040    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.447627 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.447602    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 12:35:01.447890 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.447722    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 12:35:01.447890 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.447744    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 12:35:01.447890 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.447878    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 12:35:01.448069 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.448005    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 12:35:01.448119 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.448073    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 12:35:01.448172 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.448111    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 12:35:01.448218 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.448113    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6247m\""
Jul 28 12:35:01.448218 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.448115    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 12:35:01.448374 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.448354    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 12:35:01.453381 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.453366    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:35:01.508860 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:35:01.508823    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846183c6_9edf_4a5e_b6ee_de3752e60c06.slice/crio-2fa5de095054b468d15ab5607ade46b638581e7d328cfce4044d1973c6a3cb83 WatchSource:0}: Error finding container 2fa5de095054b468d15ab5607ade46b638581e7d328cfce4044d1973c6a3cb83: Status 404 returned error can't find the container with id 2fa5de095054b468d15ab5607ade46b638581e7d328cfce4044d1973c6a3cb83
Jul 28 12:35:01.524303 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524404 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524461 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524416    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524512 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524461    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524512 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524612 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524529    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524664 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524664 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524636    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524759 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524660    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qhn\" (UniqueName: \"kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-kube-api-access-v2qhn\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524759 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524759 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524741    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524897 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524786    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-out\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.524897 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.524807    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-web-config\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.566256 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.566177    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbx9v" event={"ID":"846183c6-9edf-4a5e-b6ee-de3752e60c06","Type":"ContainerStarted","Data":"2fa5de095054b468d15ab5607ade46b638581e7d328cfce4044d1973c6a3cb83"}
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625221    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625268    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qhn\" (UniqueName: \"kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-kube-api-access-v2qhn\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625331    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625391    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-out\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625413    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-web-config\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625442    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625483    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625554    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625586    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.626902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.625615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.630467 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.630088    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.637215 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.636775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.637215 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.636870    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-out\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.637215 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.636909    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.638634 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.638572    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.641999 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.641967    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.648972 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.648913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.649397 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.649374    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.649933 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.649769    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-web-config\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.650594 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.650256    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.650594 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.650495    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.651443 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.650694    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.652115 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.651570    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qhn\" (UniqueName: \"kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-kube-api-access-v2qhn\") pod \"alertmanager-main-0\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.667142 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.667049    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6"]
Jul 28 12:35:01.693014 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.692966    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2"]
Jul 28 12:35:01.705227 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:35:01.705191    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845b2466_b87c_4d92_97ba_0f2309e87624.slice/crio-29ab0dd66a9a6eba4aeb812a5268b56d1eb40dc37230c476b68d8a15470ff1bf WatchSource:0}: Error finding container 29ab0dd66a9a6eba4aeb812a5268b56d1eb40dc37230c476b68d8a15470ff1bf: Status 404 returned error can't find the container with id 29ab0dd66a9a6eba4aeb812a5268b56d1eb40dc37230c476b68d8a15470ff1bf
Jul 28 12:35:01.756339 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.756276    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:35:01.926598 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:01.926495    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:35:01.931776 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:35:01.931743    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1579f16_b3d2_4977_a9d9_b1804f6089dd.slice/crio-a7508ee3c528bf86dfaeac4542213a096eb961e26dd9cd640aed5135186279d4 WatchSource:0}: Error finding container a7508ee3c528bf86dfaeac4542213a096eb961e26dd9cd640aed5135186279d4: Status 404 returned error can't find the container with id a7508ee3c528bf86dfaeac4542213a096eb961e26dd9cd640aed5135186279d4
Jul 28 12:35:02.353251 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.353217    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-698d9cc9fc-6f874"]
Jul 28 12:35:02.357869 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.357848    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.361771 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.361727    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-48co83nlb6641\""
Jul 28 12:35:02.362066 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.362045    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 12:35:02.362276 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.362259    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 12:35:02.362503 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.362484    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8v596\""
Jul 28 12:35:02.362735 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.362721    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 12:35:02.362972 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.362936    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 12:35:02.363084 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.363059    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 12:35:02.372291 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.372198    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-698d9cc9fc-6f874"]
Jul 28 12:35:02.435022 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.434965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-tls\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.435482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.435051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b774a01-b5c9-42a9-bc44-e2237cf95830-metrics-client-ca\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.435482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.435098    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.435482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.435175    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.435482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.435249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-grpc-tls\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.435482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.435299    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.435482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.435343    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28blx\" (UniqueName: \"kubernetes.io/projected/9b774a01-b5c9-42a9-bc44-e2237cf95830-kube-api-access-28blx\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.435482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.435388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.535845 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.535816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.535962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.535859    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-grpc-tls\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.535962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.535892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.535962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.535918    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28blx\" (UniqueName: \"kubernetes.io/projected/9b774a01-b5c9-42a9-bc44-e2237cf95830-kube-api-access-28blx\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.535962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.535947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.536195 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.536012    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-tls\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.536195 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.536059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b774a01-b5c9-42a9-bc44-e2237cf95830-metrics-client-ca\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.536195 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.536100    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.537378 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.537218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b774a01-b5c9-42a9-bc44-e2237cf95830-metrics-client-ca\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.539734 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.539364    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.539734 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.539411    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.539734 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.539571    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-grpc-tls\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.539734 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.539597    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.540707 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.540657    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-tls\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.541254 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.541231    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b774a01-b5c9-42a9-bc44-e2237cf95830-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.544082 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.544040    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28blx\" (UniqueName: \"kubernetes.io/projected/9b774a01-b5c9-42a9-bc44-e2237cf95830-kube-api-access-28blx\") pod \"thanos-querier-698d9cc9fc-6f874\" (UID: \"9b774a01-b5c9-42a9-bc44-e2237cf95830\") " pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.572601 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.572559    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69bbdccd5c-ltgjb" event={"ID":"77fac7f3-5fa0-4364-afd1-c8739b27d163","Type":"ContainerStarted","Data":"9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c"}
Jul 28 12:35:02.577412 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.576770    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f58jt" event={"ID":"560ade9a-e010-41ab-9174-54a41e8d323e","Type":"ContainerStarted","Data":"1a412127f8471817c883a45441aea0f5634770212b5e5c267f0cb974bd28a30e"}
Jul 28 12:35:02.579742 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.579672    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2" event={"ID":"845b2466-b87c-4d92-97ba-0f2309e87624","Type":"ContainerStarted","Data":"29ab0dd66a9a6eba4aeb812a5268b56d1eb40dc37230c476b68d8a15470ff1bf"}
Jul 28 12:35:02.583475 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.583378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerStarted","Data":"a7508ee3c528bf86dfaeac4542213a096eb961e26dd9cd640aed5135186279d4"}
Jul 28 12:35:02.585379 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.585347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6" event={"ID":"dc80c0af-bbf0-43f8-a9fc-f72af0510f41","Type":"ContainerStarted","Data":"8be5886dc93ddd8806323b452d0ce4ace5a7427df22bf21a40f063f73bb0114e"}
Jul 28 12:35:02.585477 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.585399    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6" event={"ID":"dc80c0af-bbf0-43f8-a9fc-f72af0510f41","Type":"ContainerStarted","Data":"2ec803c41495ddaae262efa2c2609e948679c054f51688859f74cfc605ae9df3"}
Jul 28 12:35:02.585477 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.585427    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6" event={"ID":"dc80c0af-bbf0-43f8-a9fc-f72af0510f41","Type":"ContainerStarted","Data":"5d85f8e32b4e3a6886dc0826d6a0ad17b73c767248a67454f826a9543f06b29f"}
Jul 28 12:35:02.610476 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.609846    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f58jt" podStartSLOduration=2.014232833 podStartE2EDuration="16.609827456s" totalImagesPullingTime="14.595594623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:02.609376218 +0000 UTC m=+70.824892888" watchObservedRunningTime="2026-07-28 12:35:02.609827456 +0000 UTC m=+70.825344122"
Jul 28 12:35:02.610709 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.610667    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69bbdccd5c-ltgjb" podStartSLOduration=1.5952426979999998 podStartE2EDuration="6.610653368s" totalImagesPullingTime="5.01541067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:02.591815032 +0000 UTC m=+70.807331685" watchObservedRunningTime="2026-07-28 12:35:02.610653368 +0000 UTC m=+70.826170022"
Jul 28 12:35:02.671383 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.671219    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:02.892478 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:02.892373    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-698d9cc9fc-6f874"]
Jul 28 12:35:03.593951 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:03.593168    2569 generic.go:356] "Generic (PLEG): container finished" podID="846183c6-9edf-4a5e-b6ee-de3752e60c06" containerID="dc09610eedb2fcb8b1e36148ed77ce47fa913fb450e0f3a35bc899b699fbc9cc" exitCode=0
Jul 28 12:35:03.593951 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:03.593256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbx9v" event={"ID":"846183c6-9edf-4a5e-b6ee-de3752e60c06","Type":"ContainerDied","Data":"dc09610eedb2fcb8b1e36148ed77ce47fa913fb450e0f3a35bc899b699fbc9cc"}
Jul 28 12:35:03.594913 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:03.594882    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874" event={"ID":"9b774a01-b5c9-42a9-bc44-e2237cf95830","Type":"ContainerStarted","Data":"000d8ac72f8188bf1fe61bf36618a9bcc186d61d8e36160bf60ccd45ca763b16"}
Jul 28 12:35:03.599166 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:03.599132    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jp6n" event={"ID":"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3","Type":"ContainerStarted","Data":"85156518b3ed1fd0bc8a02663a0b60fa94faee99e51f7bbe00f281fa8e401513"}
Jul 28 12:35:03.599297 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:03.599196    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jp6n" event={"ID":"afc8521c-7a71-4ec0-9e3a-53d4512d5ae3","Type":"ContainerStarted","Data":"2c43e1e2ffaf45a196c6bcd773093256cfbde0244d5124dea321c7b9efd78b19"}
Jul 28 12:35:03.626356 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:03.624800    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6jp6n" podStartSLOduration=67.31257511 podStartE2EDuration="1m11.624780965s" totalImagesPullingTime="4.312205855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:03.624072695 +0000 UTC m=+71.839589348" watchObservedRunningTime="2026-07-28 12:35:03.624780965 +0000 UTC m=+71.840297619"
Jul 28 12:35:04.622496 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.622466    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-844d945947-w9w6z"]
Jul 28 12:35:04.627700 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.627672    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.637933 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.637741    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 12:35:04.639361 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.639303    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-844d945947-w9w6z"]
Jul 28 12:35:04.699197 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.699157    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6759974dfc-8ghkv"]
Jul 28 12:35:04.702645 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.702618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.706439 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.705047    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-58fh2\""
Jul 28 12:35:04.706439 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.705430    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 12:35:04.706439 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.705702    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eflff9mbrer2b\""
Jul 28 12:35:04.706439 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.705951    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 12:35:04.706439 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.706153    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 12:35:04.706439 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.706338    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 12:35:04.713412 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.713388    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6759974dfc-8ghkv"]
Jul 28 12:35:04.758593 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.758561    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-serving-cert\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.758744 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.758632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9qc\" (UniqueName: \"kubernetes.io/projected/d68d448f-212b-4487-85c8-a628794a75d3-kube-api-access-gq9qc\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.758744 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.758668    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-trusted-ca-bundle\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.758871 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.758739    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-oauth-serving-cert\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.758871 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.758795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-oauth-config\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.758976 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.758866    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-console-config\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.758976 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.758914    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-service-ca\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.859627 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-service-ca\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.859807 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859640    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sgtd\" (UniqueName: \"kubernetes.io/projected/5773d6fb-afa0-4db1-8105-ce4d08e50a59-kube-api-access-7sgtd\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.859807 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-serving-cert\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.859807 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5773d6fb-afa0-4db1-8105-ce4d08e50a59-secret-metrics-server-client-certs\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.859807 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859724    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5773d6fb-afa0-4db1-8105-ce4d08e50a59-audit-log\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.859807 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5773d6fb-afa0-4db1-8105-ce4d08e50a59-metrics-server-audit-profiles\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.859807 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5773d6fb-afa0-4db1-8105-ce4d08e50a59-secret-metrics-server-tls\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.860135 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9qc\" (UniqueName: \"kubernetes.io/projected/d68d448f-212b-4487-85c8-a628794a75d3-kube-api-access-gq9qc\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.860135 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859857    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5773d6fb-afa0-4db1-8105-ce4d08e50a59-client-ca-bundle\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.860135 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859885    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-trusted-ca-bundle\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.860135 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859918    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-oauth-serving-cert\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.860135 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.859962    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-oauth-config\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.860135 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.860036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-console-config\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.860135 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.860061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5773d6fb-afa0-4db1-8105-ce4d08e50a59-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.861518 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.860857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-service-ca\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.861518 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.861471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-console-config\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.861755 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.861730    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-trusted-ca-bundle\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.861829 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.861734    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-oauth-serving-cert\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.864330 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.864287    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-oauth-config\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.865508 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.865484    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-serving-cert\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.871282 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.871236    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9qc\" (UniqueName: \"kubernetes.io/projected/d68d448f-212b-4487-85c8-a628794a75d3-kube-api-access-gq9qc\") pod \"console-844d945947-w9w6z\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") " pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.945113 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.945031    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:04.960618 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.960592    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5773d6fb-afa0-4db1-8105-ce4d08e50a59-secret-metrics-server-client-certs\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.960738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.960628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5773d6fb-afa0-4db1-8105-ce4d08e50a59-audit-log\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.960738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.960654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5773d6fb-afa0-4db1-8105-ce4d08e50a59-metrics-server-audit-profiles\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.960738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.960692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5773d6fb-afa0-4db1-8105-ce4d08e50a59-secret-metrics-server-tls\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.960738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.960730    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5773d6fb-afa0-4db1-8105-ce4d08e50a59-client-ca-bundle\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.960983 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.960796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5773d6fb-afa0-4db1-8105-ce4d08e50a59-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.960983 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.960844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sgtd\" (UniqueName: \"kubernetes.io/projected/5773d6fb-afa0-4db1-8105-ce4d08e50a59-kube-api-access-7sgtd\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.961129 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.961008    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5773d6fb-afa0-4db1-8105-ce4d08e50a59-audit-log\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.961747 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.961721    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5773d6fb-afa0-4db1-8105-ce4d08e50a59-metrics-server-audit-profiles\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.961842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.961815    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5773d6fb-afa0-4db1-8105-ce4d08e50a59-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.964761 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.964717    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5773d6fb-afa0-4db1-8105-ce4d08e50a59-secret-metrics-server-tls\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.964761 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.964717    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5773d6fb-afa0-4db1-8105-ce4d08e50a59-client-ca-bundle\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.964911 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.964789    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5773d6fb-afa0-4db1-8105-ce4d08e50a59-secret-metrics-server-client-certs\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:04.969069 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:04.968980    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sgtd\" (UniqueName: \"kubernetes.io/projected/5773d6fb-afa0-4db1-8105-ce4d08e50a59-kube-api-access-7sgtd\") pod \"metrics-server-6759974dfc-8ghkv\" (UID: \"5773d6fb-afa0-4db1-8105-ce4d08e50a59\") " pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:05.020150 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.020087    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:05.143889 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.143853    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"]
Jul 28 12:35:05.147557 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.147534    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:05.150433 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.150044    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cjdw5\""
Jul 28 12:35:05.150433 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.150287    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 12:35:05.154964 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.154913    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"]
Jul 28 12:35:05.263910 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.263874    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/274ac08e-e89c-47f6-8236-34e87fd7f597-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-m2czb\" (UID: \"274ac08e-e89c-47f6-8236-34e87fd7f597\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:05.264107 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.264055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trn6d\" (UniqueName: \"kubernetes.io/projected/274ac08e-e89c-47f6-8236-34e87fd7f597-kube-api-access-trn6d\") pod \"monitoring-plugin-68b7fc9574-m2czb\" (UID: \"274ac08e-e89c-47f6-8236-34e87fd7f597\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:05.365501 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.365465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trn6d\" (UniqueName: \"kubernetes.io/projected/274ac08e-e89c-47f6-8236-34e87fd7f597-kube-api-access-trn6d\") pod \"monitoring-plugin-68b7fc9574-m2czb\" (UID: \"274ac08e-e89c-47f6-8236-34e87fd7f597\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:05.365702 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.365525    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/274ac08e-e89c-47f6-8236-34e87fd7f597-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-m2czb\" (UID: \"274ac08e-e89c-47f6-8236-34e87fd7f597\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:05.365702 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:35:05.365656    2569 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 12:35:05.365822 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:35:05.365725    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/274ac08e-e89c-47f6-8236-34e87fd7f597-monitoring-plugin-cert podName:274ac08e-e89c-47f6-8236-34e87fd7f597 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:05.865703376 +0000 UTC m=+74.081220012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/274ac08e-e89c-47f6-8236-34e87fd7f597-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-m2czb" (UID: "274ac08e-e89c-47f6-8236-34e87fd7f597") : secret "monitoring-plugin-cert" not found
Jul 28 12:35:05.376130 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.376100    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trn6d\" (UniqueName: \"kubernetes.io/projected/274ac08e-e89c-47f6-8236-34e87fd7f597-kube-api-access-trn6d\") pod \"monitoring-plugin-68b7fc9574-m2czb\" (UID: \"274ac08e-e89c-47f6-8236-34e87fd7f597\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:05.870767 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.870724    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/274ac08e-e89c-47f6-8236-34e87fd7f597-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-m2czb\" (UID: \"274ac08e-e89c-47f6-8236-34e87fd7f597\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:05.873523 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:05.873496    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/274ac08e-e89c-47f6-8236-34e87fd7f597-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-m2czb\" (UID: \"274ac08e-e89c-47f6-8236-34e87fd7f597\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:06.064113 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:06.064073    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:06.361473 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:06.361430    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:35:06.361640 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:06.361495    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:35:06.366896 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:06.366871    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:35:06.618684 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:06.618600    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:35:07.182450 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:07.182415    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"]
Jul 28 12:35:07.409376 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:07.409253    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-844d945947-w9w6z"]
Jul 28 12:35:07.412508 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:07.412413    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6759974dfc-8ghkv"]
Jul 28 12:35:07.624199 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:35:07.624163    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68d448f_212b_4487_85c8_a628794a75d3.slice/crio-5e17121e29c4adf342d895f70d71e27a8934b4f7cd2cfd77c53fd30b701f153a WatchSource:0}: Error finding container 5e17121e29c4adf342d895f70d71e27a8934b4f7cd2cfd77c53fd30b701f153a: Status 404 returned error can't find the container with id 5e17121e29c4adf342d895f70d71e27a8934b4f7cd2cfd77c53fd30b701f153a
Jul 28 12:35:07.624663 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:35:07.624639    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274ac08e_e89c_47f6_8236_34e87fd7f597.slice/crio-b8703696c2c5272db809787919cbe624f759487648d105919e7864036589387b WatchSource:0}: Error finding container b8703696c2c5272db809787919cbe624f759487648d105919e7864036589387b: Status 404 returned error can't find the container with id b8703696c2c5272db809787919cbe624f759487648d105919e7864036589387b
Jul 28 12:35:07.625620 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:35:07.625600    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5773d6fb_afa0_4db1_8105_ce4d08e50a59.slice/crio-0cd70b80d2016e1e1a51d87fe1f65841cdd1f662d67070183973be1fab04f51e WatchSource:0}: Error finding container 0cd70b80d2016e1e1a51d87fe1f65841cdd1f662d67070183973be1fab04f51e: Status 404 returned error can't find the container with id 0cd70b80d2016e1e1a51d87fe1f65841cdd1f662d67070183973be1fab04f51e
Jul 28 12:35:08.625750 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.625708    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kffxs" event={"ID":"a4544365-5759-41c2-8b2a-0f96b508e3d5","Type":"ContainerStarted","Data":"e29dc6d56ba5f974185299abb912fe19422dd491c9426537eb411d34e3e3c797"}
Jul 28 12:35:08.626236 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.625803    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:35:08.628663 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.628121    2569 generic.go:356] "Generic (PLEG): container finished" podID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerID="7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885" exitCode=0
Jul 28 12:35:08.628663 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.628169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerDied","Data":"7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885"}
Jul 28 12:35:08.632734 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.632689    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6" event={"ID":"dc80c0af-bbf0-43f8-a9fc-f72af0510f41","Type":"ContainerStarted","Data":"7a5ba6f791e9fe9c865b7f5a51c18a3d9e5d81164dd16c5f66c40f57a9ae987d"}
Jul 28 12:35:08.635665 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.635637    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv" event={"ID":"5773d6fb-afa0-4db1-8105-ce4d08e50a59","Type":"ContainerStarted","Data":"0cd70b80d2016e1e1a51d87fe1f65841cdd1f662d67070183973be1fab04f51e"}
Jul 28 12:35:08.638325 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.637873    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdsfj" event={"ID":"79d7959e-9636-495a-930c-a09f71fb6c57","Type":"ContainerStarted","Data":"804292c754222884111a89a83c8e04b25a6ddfc678b37bbeeb075d46e4ba1743"}
Jul 28 12:35:08.642421 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.642382    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kffxs" podStartSLOduration=68.070004754 podStartE2EDuration="1m16.642366292s" totalImagesPullingTime="8.572361538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:08.641554758 +0000 UTC m=+76.857078554" watchObservedRunningTime="2026-07-28 12:35:08.642366292 +0000 UTC m=+76.857882943"
Jul 28 12:35:08.643107 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.643080    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb" event={"ID":"274ac08e-e89c-47f6-8236-34e87fd7f597","Type":"ContainerStarted","Data":"b8703696c2c5272db809787919cbe624f759487648d105919e7864036589387b"}
Jul 28 12:35:08.646793 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.646769    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbx9v" event={"ID":"846183c6-9edf-4a5e-b6ee-de3752e60c06","Type":"ContainerStarted","Data":"b0ceb83e70a9b281392014093124dd65be24005331072966e6aeef351890846c"}
Jul 28 12:35:08.646895 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.646815    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbx9v" event={"ID":"846183c6-9edf-4a5e-b6ee-de3752e60c06","Type":"ContainerStarted","Data":"af0b61e4220c4378cfac5d0541a95d07ede251ca38c8cf3a03c082f081bac436"}
Jul 28 12:35:08.650356 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.650141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2" event={"ID":"845b2466-b87c-4d92-97ba-0f2309e87624","Type":"ContainerStarted","Data":"43722ee7a7b792bfacf3446c1f6bb3bddeb211f49f7ec8205d5e589e4cd0bf42"}
Jul 28 12:35:08.650356 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.650190    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2" event={"ID":"845b2466-b87c-4d92-97ba-0f2309e87624","Type":"ContainerStarted","Data":"32a08f64c6f72761a5e818293243f3d5cbe05f9cb9ebc8dc65867e4a97f50333"}
Jul 28 12:35:08.650356 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.650218    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2" event={"ID":"845b2466-b87c-4d92-97ba-0f2309e87624","Type":"ContainerStarted","Data":"8f69a6ab74b05cfaf54bad234c4f09df1127a302780df530009040021b38d996"}
Jul 28 12:35:08.653260 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.653106    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874" event={"ID":"9b774a01-b5c9-42a9-bc44-e2237cf95830","Type":"ContainerStarted","Data":"6babf4745b8e3fdb77cb40e824a59b4c364c9705f306eadfaf049357ca8698ee"}
Jul 28 12:35:08.653260 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.653175    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874" event={"ID":"9b774a01-b5c9-42a9-bc44-e2237cf95830","Type":"ContainerStarted","Data":"1dcc0e2f518db4794f397499fe8ba30eda6a78f6370b54364e82a4874a18159e"}
Jul 28 12:35:08.653260 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.653214    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874" event={"ID":"9b774a01-b5c9-42a9-bc44-e2237cf95830","Type":"ContainerStarted","Data":"e9f6cb66ed9d1dcb099e539ca6cd236c9709c1423bc6be4eca4d65286b77ec23"}
Jul 28 12:35:08.656432 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.656407    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844d945947-w9w6z" event={"ID":"d68d448f-212b-4487-85c8-a628794a75d3","Type":"ContainerStarted","Data":"838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee"}
Jul 28 12:35:08.656499 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.656450    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844d945947-w9w6z" event={"ID":"d68d448f-212b-4487-85c8-a628794a75d3","Type":"ContainerStarted","Data":"5e17121e29c4adf342d895f70d71e27a8934b4f7cd2cfd77c53fd30b701f153a"}
Jul 28 12:35:08.683151 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.683093    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xdsfj" podStartSLOduration=67.305425003 podStartE2EDuration="1m16.683072492s" totalImagesPullingTime="9.377647489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:08.681331194 +0000 UTC m=+76.896847847" watchObservedRunningTime="2026-07-28 12:35:08.683072492 +0000 UTC m=+76.898589144"
Jul 28 12:35:08.701403 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.701021    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-j6bx6" podStartSLOduration=3.498777022 podStartE2EDuration="8.700999916s" totalImagesPullingTime="5.202222894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:08.699062705 +0000 UTC m=+76.914579358" watchObservedRunningTime="2026-07-28 12:35:08.700999916 +0000 UTC m=+76.916516560"
Jul 28 12:35:08.720124 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.719652    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-g5dt2" podStartSLOduration=3.39223126 podStartE2EDuration="8.719633989s" totalImagesPullingTime="5.327402729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:08.718911885 +0000 UTC m=+76.934428550" watchObservedRunningTime="2026-07-28 12:35:08.719633989 +0000 UTC m=+76.935150643"
Jul 28 12:35:08.751277 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.751152    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dbx9v" podStartSLOduration=7.562619335 podStartE2EDuration="8.751130676s" totalImagesPullingTime="1.065736157s" totalInitContainerRuntime="122.775184ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:08.743171167 +0000 UTC m=+76.958687821" watchObservedRunningTime="2026-07-28 12:35:08.751130676 +0000 UTC m=+76.966647329"
Jul 28 12:35:08.772705 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:08.772645    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-844d945947-w9w6z" podStartSLOduration=4.772622062 podStartE2EDuration="4.772622062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:08.770795489 +0000 UTC m=+76.986312144" watchObservedRunningTime="2026-07-28 12:35:08.772622062 +0000 UTC m=+76.988138716"
Jul 28 12:35:09.505673 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:09.505641    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dc9fbdd49-tlf2w"
Jul 28 12:35:10.442397 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.442365    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-844d945947-w9w6z"]
Jul 28 12:35:10.485375 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.485335    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-db8f99589-tqpzl"]
Jul 28 12:35:10.489980 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.489956    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.501178 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.500647    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db8f99589-tqpzl"]
Jul 28 12:35:10.624951 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.624917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-trusted-ca-bundle\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.625122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.624963    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-oauth-config\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.625122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.625004    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-config\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.625122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.625063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfm2m\" (UniqueName: \"kubernetes.io/projected/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-kube-api-access-vfm2m\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.625122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.625097    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-serving-cert\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.625122 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.625124    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-service-ca\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.625303 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.625138    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-oauth-serving-cert\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.668967 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.668929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874" event={"ID":"9b774a01-b5c9-42a9-bc44-e2237cf95830","Type":"ContainerStarted","Data":"71c28b3c67a4c03cb04051feefa8db287e501527cd318f07df1aa7bff2396501"}
Jul 28 12:35:10.669149 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.669009    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874" event={"ID":"9b774a01-b5c9-42a9-bc44-e2237cf95830","Type":"ContainerStarted","Data":"2e2c1b874c06eaa004d6d009905ce85d46eea719fd5aa9909684c6f75cb84e3c"}
Jul 28 12:35:10.669149 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.669061    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:10.669149 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.669089    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874" event={"ID":"9b774a01-b5c9-42a9-bc44-e2237cf95830","Type":"ContainerStarted","Data":"0409dea4faaf74c7e2bbd91f500981c07a94762201c50ed8b85ec77aaa7a3976"}
Jul 28 12:35:10.670371 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.670341    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv" event={"ID":"5773d6fb-afa0-4db1-8105-ce4d08e50a59","Type":"ContainerStarted","Data":"9f921166da3abdfe5e4b890c23ce4ce7875c99757259388b666de2f61762aa07"}
Jul 28 12:35:10.671768 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.671740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb" event={"ID":"274ac08e-e89c-47f6-8236-34e87fd7f597","Type":"ContainerStarted","Data":"a6efe1fd9a443f77c8bdfd423dc313a47ddb08be63e7f45fea992150798a12d0"}
Jul 28 12:35:10.671960 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.671941    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:10.677210 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.677192    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb"
Jul 28 12:35:10.693707 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.693622    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874" podStartSLOduration=2.297272051 podStartE2EDuration="8.693603226s" totalImagesPullingTime="6.396331175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:10.692874066 +0000 UTC m=+78.908390725" watchObservedRunningTime="2026-07-28 12:35:10.693603226 +0000 UTC m=+78.909119878"
Jul 28 12:35:10.725981 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.725956    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-serving-cert\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.726105 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.726019    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-service-ca\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.726105 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.726046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-oauth-serving-cert\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.726105 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.726087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-trusted-ca-bundle\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.726274 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.726127    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-oauth-config\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.726274 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.726158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-config\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.726274 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.726200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfm2m\" (UniqueName: \"kubernetes.io/projected/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-kube-api-access-vfm2m\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.727153 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.727127    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-service-ca\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.727502 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.727481    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-oauth-serving-cert\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.727560 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.727489    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-config\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.728034 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.728016    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-trusted-ca-bundle\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.728474 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.728435    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m2czb" podStartSLOduration=3.750370332 podStartE2EDuration="5.728421933s" totalImagesPullingTime="1.978051601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:10.727286551 +0000 UTC m=+78.942803204" watchObservedRunningTime="2026-07-28 12:35:10.728421933 +0000 UTC m=+78.943938585"
Jul 28 12:35:10.728569 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.728537    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-oauth-config\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.728725 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.728707    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-serving-cert\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.739034 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.739011    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfm2m\" (UniqueName: \"kubernetes.io/projected/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-kube-api-access-vfm2m\") pod \"console-db8f99589-tqpzl\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") " pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.749702 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.749647    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv" podStartSLOduration=4.772902194 podStartE2EDuration="6.749629148s" totalImagesPullingTime="1.976726954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:10.748707537 +0000 UTC m=+78.964224193" watchObservedRunningTime="2026-07-28 12:35:10.749629148 +0000 UTC m=+78.965145801"
Jul 28 12:35:10.805893 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.805863    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:10.968262 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:10.968205    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db8f99589-tqpzl"]
Jul 28 12:35:10.971060 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:35:10.971036    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e89fc8_8390_4a8f_b69e_6b7c948ec15c.slice/crio-a31097886fddb12d0405ebf90d60b2772c3b6857bd3e27eb1a68e1911dc74de3 WatchSource:0}: Error finding container a31097886fddb12d0405ebf90d60b2772c3b6857bd3e27eb1a68e1911dc74de3: Status 404 returned error can't find the container with id a31097886fddb12d0405ebf90d60b2772c3b6857bd3e27eb1a68e1911dc74de3
Jul 28 12:35:11.678941 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:11.678904    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerStarted","Data":"9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde"}
Jul 28 12:35:11.679364 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:11.678975    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerStarted","Data":"703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747"}
Jul 28 12:35:11.679364 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:11.679041    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerStarted","Data":"4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4"}
Jul 28 12:35:11.679364 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:11.679084    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerStarted","Data":"5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38"}
Jul 28 12:35:11.679364 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:11.679108    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerStarted","Data":"84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98"}
Jul 28 12:35:11.679364 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:11.679133    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerStarted","Data":"30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03"}
Jul 28 12:35:11.680575 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:11.680550    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db8f99589-tqpzl" event={"ID":"26e89fc8-8390-4a8f-b69e-6b7c948ec15c","Type":"ContainerStarted","Data":"8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5"}
Jul 28 12:35:11.680678 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:11.680593    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db8f99589-tqpzl" event={"ID":"26e89fc8-8390-4a8f-b69e-6b7c948ec15c","Type":"ContainerStarted","Data":"a31097886fddb12d0405ebf90d60b2772c3b6857bd3e27eb1a68e1911dc74de3"}
Jul 28 12:35:11.707303 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:11.707252    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.537375768 podStartE2EDuration="10.707231993s" totalImagesPullingTime="7.146651717s" totalInitContainerRuntime="23.204508ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:11.705620945 +0000 UTC m=+79.921137656" watchObservedRunningTime="2026-07-28 12:35:11.707231993 +0000 UTC m=+79.922748648"
Jul 28 12:35:11.724007 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:11.723929    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-db8f99589-tqpzl" podStartSLOduration=1.7239074840000002 podStartE2EDuration="1.723907484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:11.72309918 +0000 UTC m=+79.938615833" watchObservedRunningTime="2026-07-28 12:35:11.723907484 +0000 UTC m=+79.939424137"
Jul 28 12:35:14.945601 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:14.945485    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:16.686885 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:16.686855    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-698d9cc9fc-6f874"
Jul 28 12:35:20.806855 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:20.806823    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:20.807314 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:20.806884    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:20.811964 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:20.811942    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:21.725451 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:21.725418    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:35:21.789316 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:21.789275    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69bbdccd5c-ltgjb"]
Jul 28 12:35:25.021101 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:25.021062    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:25.021490 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:25.021119    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:35.690892 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:35.690851    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-844d945947-w9w6z" podUID="d68d448f-212b-4487-85c8-a628794a75d3" containerName="console" containerID="cri-o://838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee" gracePeriod=15
Jul 28 12:35:36.468192 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.468165    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:36.543849 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.543822    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-trusted-ca-bundle\") pod \"d68d448f-212b-4487-85c8-a628794a75d3\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") "
Jul 28 12:35:36.544014 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.543871    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq9qc\" (UniqueName: \"kubernetes.io/projected/d68d448f-212b-4487-85c8-a628794a75d3-kube-api-access-gq9qc\") pod \"d68d448f-212b-4487-85c8-a628794a75d3\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") "
Jul 28 12:35:36.544014 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.543888    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-console-config\") pod \"d68d448f-212b-4487-85c8-a628794a75d3\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") "
Jul 28 12:35:36.544119 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.544049    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-serving-cert\") pod \"d68d448f-212b-4487-85c8-a628794a75d3\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") "
Jul 28 12:35:36.544119 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.544088    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-service-ca\") pod \"d68d448f-212b-4487-85c8-a628794a75d3\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") "
Jul 28 12:35:36.544221 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.544154    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-oauth-serving-cert\") pod \"d68d448f-212b-4487-85c8-a628794a75d3\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") "
Jul 28 12:35:36.544221 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.544183    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-oauth-config\") pod \"d68d448f-212b-4487-85c8-a628794a75d3\" (UID: \"d68d448f-212b-4487-85c8-a628794a75d3\") "
Jul 28 12:35:36.544323 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.544216    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d68d448f-212b-4487-85c8-a628794a75d3" (UID: "d68d448f-212b-4487-85c8-a628794a75d3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:35:36.544323 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.544251    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-console-config" (OuterVolumeSpecName: "console-config") pod "d68d448f-212b-4487-85c8-a628794a75d3" (UID: "d68d448f-212b-4487-85c8-a628794a75d3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:35:36.544485 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.544451    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-trusted-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:36.544485 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.544458    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-service-ca" (OuterVolumeSpecName: "service-ca") pod "d68d448f-212b-4487-85c8-a628794a75d3" (UID: "d68d448f-212b-4487-85c8-a628794a75d3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:35:36.544485 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.544487    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-console-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:36.544674 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.544525    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d68d448f-212b-4487-85c8-a628794a75d3" (UID: "d68d448f-212b-4487-85c8-a628794a75d3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:35:36.546428 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.546408    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d68d448f-212b-4487-85c8-a628794a75d3" (UID: "d68d448f-212b-4487-85c8-a628794a75d3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:35:36.546685 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.546669    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d68d448f-212b-4487-85c8-a628794a75d3" (UID: "d68d448f-212b-4487-85c8-a628794a75d3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:35:36.546747 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.546722    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68d448f-212b-4487-85c8-a628794a75d3-kube-api-access-gq9qc" (OuterVolumeSpecName: "kube-api-access-gq9qc") pod "d68d448f-212b-4487-85c8-a628794a75d3" (UID: "d68d448f-212b-4487-85c8-a628794a75d3"). InnerVolumeSpecName "kube-api-access-gq9qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:35:36.645120 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.645047    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-oauth-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:36.645120 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.645069    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-oauth-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:36.645120 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.645078    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9qc\" (UniqueName: \"kubernetes.io/projected/d68d448f-212b-4487-85c8-a628794a75d3-kube-api-access-gq9qc\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:36.645120 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.645086    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d68d448f-212b-4487-85c8-a628794a75d3-console-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:36.645120 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.645096    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d68d448f-212b-4487-85c8-a628794a75d3-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:36.782191 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.782155    2569 generic.go:356] "Generic (PLEG): container finished" podID="d68d448f-212b-4487-85c8-a628794a75d3" containerID="838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee" exitCode=0
Jul 28 12:35:36.782667 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.782200    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844d945947-w9w6z"
Jul 28 12:35:36.782667 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.782240    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844d945947-w9w6z" event={"ID":"d68d448f-212b-4487-85c8-a628794a75d3","Type":"ContainerDied","Data":"838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee"}
Jul 28 12:35:36.782667 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.782295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844d945947-w9w6z" event={"ID":"d68d448f-212b-4487-85c8-a628794a75d3","Type":"ContainerDied","Data":"5e17121e29c4adf342d895f70d71e27a8934b4f7cd2cfd77c53fd30b701f153a"}
Jul 28 12:35:36.782667 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.782327    2569 scope.go:117] "RemoveContainer" containerID="838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee"
Jul 28 12:35:36.791788 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.791766    2569 scope.go:117] "RemoveContainer" containerID="838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee"
Jul 28 12:35:36.792076 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:35:36.792053    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee\": container with ID starting with 838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee not found: ID does not exist" containerID="838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee"
Jul 28 12:35:36.792163 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.792083    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee"} err="failed to get container status \"838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee\": rpc error: code = NotFound desc = could not find container \"838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee\": container with ID starting with 838c16e025962be9a7ebfaf2b5b13eaf951b54497228f5485477d469a2076cee not found: ID does not exist"
Jul 28 12:35:36.803259 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.803236    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-844d945947-w9w6z"]
Jul 28 12:35:36.806093 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:36.806075    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-844d945947-w9w6z"]
Jul 28 12:35:38.250854 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:38.250822    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d68d448f-212b-4487-85c8-a628794a75d3" path="/var/lib/kubelet/pods/d68d448f-212b-4487-85c8-a628794a75d3/volumes"
Jul 28 12:35:39.665680 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:39.665648    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kffxs"
Jul 28 12:35:45.026138 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:45.026107    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:45.030005 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:45.029970    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6759974dfc-8ghkv"
Jul 28 12:35:46.814716 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:46.814667    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69bbdccd5c-ltgjb" podUID="77fac7f3-5fa0-4364-afd1-c8739b27d163" containerName="console" containerID="cri-o://9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c" gracePeriod=15
Jul 28 12:35:47.053374 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.053348    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:35:47.136544 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.136460    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-config\") pod \"77fac7f3-5fa0-4364-afd1-c8739b27d163\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") "
Jul 28 12:35:47.136544 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.136523    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-oauth-config\") pod \"77fac7f3-5fa0-4364-afd1-c8739b27d163\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") "
Jul 28 12:35:47.136718 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.136567    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khvvz\" (UniqueName: \"kubernetes.io/projected/77fac7f3-5fa0-4364-afd1-c8739b27d163-kube-api-access-khvvz\") pod \"77fac7f3-5fa0-4364-afd1-c8739b27d163\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") "
Jul 28 12:35:47.136718 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.136593    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-oauth-serving-cert\") pod \"77fac7f3-5fa0-4364-afd1-c8739b27d163\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") "
Jul 28 12:35:47.136718 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.136656    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-service-ca\") pod \"77fac7f3-5fa0-4364-afd1-c8739b27d163\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") "
Jul 28 12:35:47.136718 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.136689    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-serving-cert\") pod \"77fac7f3-5fa0-4364-afd1-c8739b27d163\" (UID: \"77fac7f3-5fa0-4364-afd1-c8739b27d163\") "
Jul 28 12:35:47.136983 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.136959    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-config" (OuterVolumeSpecName: "console-config") pod "77fac7f3-5fa0-4364-afd1-c8739b27d163" (UID: "77fac7f3-5fa0-4364-afd1-c8739b27d163"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:35:47.137074 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.137055    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "77fac7f3-5fa0-4364-afd1-c8739b27d163" (UID: "77fac7f3-5fa0-4364-afd1-c8739b27d163"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:35:47.137128 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.137080    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-service-ca" (OuterVolumeSpecName: "service-ca") pod "77fac7f3-5fa0-4364-afd1-c8739b27d163" (UID: "77fac7f3-5fa0-4364-afd1-c8739b27d163"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:35:47.138858 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.138833    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "77fac7f3-5fa0-4364-afd1-c8739b27d163" (UID: "77fac7f3-5fa0-4364-afd1-c8739b27d163"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:35:47.138962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.138836    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fac7f3-5fa0-4364-afd1-c8739b27d163-kube-api-access-khvvz" (OuterVolumeSpecName: "kube-api-access-khvvz") pod "77fac7f3-5fa0-4364-afd1-c8739b27d163" (UID: "77fac7f3-5fa0-4364-afd1-c8739b27d163"). InnerVolumeSpecName "kube-api-access-khvvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:35:47.138962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.138908    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "77fac7f3-5fa0-4364-afd1-c8739b27d163" (UID: "77fac7f3-5fa0-4364-afd1-c8739b27d163"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:35:47.238309 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.238273    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:47.238309 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.238305    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-oauth-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:47.238309 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.238314    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khvvz\" (UniqueName: \"kubernetes.io/projected/77fac7f3-5fa0-4364-afd1-c8739b27d163-kube-api-access-khvvz\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:47.238533 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.238325    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-oauth-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:47.238533 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.238335    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77fac7f3-5fa0-4364-afd1-c8739b27d163-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:47.238533 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.238344    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77fac7f3-5fa0-4364-afd1-c8739b27d163-console-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:35:47.826839 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.826807    2569 generic.go:356] "Generic (PLEG): container finished" podID="77fac7f3-5fa0-4364-afd1-c8739b27d163" containerID="9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c" exitCode=0
Jul 28 12:35:47.827264 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.826876    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69bbdccd5c-ltgjb"
Jul 28 12:35:47.827264 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.826884    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69bbdccd5c-ltgjb" event={"ID":"77fac7f3-5fa0-4364-afd1-c8739b27d163","Type":"ContainerDied","Data":"9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c"}
Jul 28 12:35:47.827264 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.826933    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69bbdccd5c-ltgjb" event={"ID":"77fac7f3-5fa0-4364-afd1-c8739b27d163","Type":"ContainerDied","Data":"145b9af5871f35ce5bebd96e892b09d2e67e8b22eb997a516f12dd41da5a2adf"}
Jul 28 12:35:47.827264 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.826953    2569 scope.go:117] "RemoveContainer" containerID="9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c"
Jul 28 12:35:47.838851 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.838833    2569 scope.go:117] "RemoveContainer" containerID="9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c"
Jul 28 12:35:47.839188 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:35:47.839166    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c\": container with ID starting with 9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c not found: ID does not exist" containerID="9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c"
Jul 28 12:35:47.839242 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.839197    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c"} err="failed to get container status \"9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c\": rpc error: code = NotFound desc = could not find container \"9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c\": container with ID starting with 9bf87dc09afb0045d52de8459db302576bae1dff0ec2a6f71d38469ce04a578c not found: ID does not exist"
Jul 28 12:35:47.852467 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.852446    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69bbdccd5c-ltgjb"]
Jul 28 12:35:47.856354 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:47.856335    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69bbdccd5c-ltgjb"]
Jul 28 12:35:48.251110 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:35:48.251081    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77fac7f3-5fa0-4364-afd1-c8739b27d163" path="/var/lib/kubelet/pods/77fac7f3-5fa0-4364-afd1-c8739b27d163/volumes"
Jul 28 12:36:20.667510 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.667472    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:20.668305 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.667761    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="alertmanager" containerID="cri-o://30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03" gracePeriod=120
Jul 28 12:36:20.668305 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.667845    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy-metric" containerID="cri-o://703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747" gracePeriod=120
Jul 28 12:36:20.668305 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.667880    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="prom-label-proxy" containerID="cri-o://9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde" gracePeriod=120
Jul 28 12:36:20.668305 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.667907    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy-web" containerID="cri-o://5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38" gracePeriod=120
Jul 28 12:36:20.668305 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.667926    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="config-reloader" containerID="cri-o://84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98" gracePeriod=120
Jul 28 12:36:20.668305 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.667939    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy" containerID="cri-o://4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4" gracePeriod=120
Jul 28 12:36:20.964725 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.964689    2569 generic.go:356] "Generic (PLEG): container finished" podID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerID="9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde" exitCode=0
Jul 28 12:36:20.964725 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.964720    2569 generic.go:356] "Generic (PLEG): container finished" podID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerID="4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4" exitCode=0
Jul 28 12:36:20.964725 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.964726    2569 generic.go:356] "Generic (PLEG): container finished" podID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerID="84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98" exitCode=0
Jul 28 12:36:20.964940 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.964733    2569 generic.go:356] "Generic (PLEG): container finished" podID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerID="30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03" exitCode=0
Jul 28 12:36:20.964940 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.964755    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerDied","Data":"9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde"}
Jul 28 12:36:20.964940 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.964818    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerDied","Data":"4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4"}
Jul 28 12:36:20.964940 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.964845    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerDied","Data":"84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98"}
Jul 28 12:36:20.964940 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:20.964869    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerDied","Data":"30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03"}
Jul 28 12:36:21.913892 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.913866    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:21.971621 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.971545    2569 generic.go:356] "Generic (PLEG): container finished" podID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerID="703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747" exitCode=0
Jul 28 12:36:21.971621 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.971576    2569 generic.go:356] "Generic (PLEG): container finished" podID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerID="5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38" exitCode=0
Jul 28 12:36:21.971776 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.971622    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:21.971776 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.971619    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerDied","Data":"703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747"}
Jul 28 12:36:21.971776 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.971770    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerDied","Data":"5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38"}
Jul 28 12:36:21.971868 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.971800    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1579f16-b3d2-4977-a9d9-b1804f6089dd","Type":"ContainerDied","Data":"a7508ee3c528bf86dfaeac4542213a096eb961e26dd9cd640aed5135186279d4"}
Jul 28 12:36:21.971868 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.971839    2569 scope.go:117] "RemoveContainer" containerID="9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde"
Jul 28 12:36:21.979728 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.979710    2569 scope.go:117] "RemoveContainer" containerID="703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747"
Jul 28 12:36:21.986608 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.986585    2569 scope.go:117] "RemoveContainer" containerID="4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4"
Jul 28 12:36:21.992976 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.992956    2569 scope.go:117] "RemoveContainer" containerID="5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38"
Jul 28 12:36:21.999336 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:21.999319    2569 scope.go:117] "RemoveContainer" containerID="84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98"
Jul 28 12:36:22.005884 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.005870    2569 scope.go:117] "RemoveContainer" containerID="30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03"
Jul 28 12:36:22.012357 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.012342    2569 scope.go:117] "RemoveContainer" containerID="7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885"
Jul 28 12:36:22.018730 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.018715    2569 scope.go:117] "RemoveContainer" containerID="9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde"
Jul 28 12:36:22.018966 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:36:22.018946    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde\": container with ID starting with 9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde not found: ID does not exist" containerID="9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde"
Jul 28 12:36:22.019077 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.018977    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde"} err="failed to get container status \"9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde\": rpc error: code = NotFound desc = could not find container \"9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde\": container with ID starting with 9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde not found: ID does not exist"
Jul 28 12:36:22.019077 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019021    2569 scope.go:117] "RemoveContainer" containerID="703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747"
Jul 28 12:36:22.019077 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019053    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-out\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019244 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019086    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-tls-assets\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019244 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019122    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019244 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019149    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2qhn\" (UniqueName: \"kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-kube-api-access-v2qhn\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019244 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019181    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019244 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019228    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-web-config\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019532 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019263    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-cluster-tls-config\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019532 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:36:22.019261    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747\": container with ID starting with 703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747 not found: ID does not exist" containerID="703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747"
Jul 28 12:36:22.019532 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019300    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-metrics-client-ca\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019532 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019294    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747"} err="failed to get container status \"703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747\": rpc error: code = NotFound desc = could not find container \"703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747\": container with ID starting with 703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747 not found: ID does not exist"
Jul 28 12:36:22.019532 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019318    2569 scope.go:117] "RemoveContainer" containerID="4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4"
Jul 28 12:36:22.019532 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019327    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-main-db\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019532 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019375    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-main-tls\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019532 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019427    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-trusted-ca-bundle\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019532 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019458    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-volume\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.019532 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019482    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-web\") pod \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\" (UID: \"d1579f16-b3d2-4977-a9d9-b1804f6089dd\") "
Jul 28 12:36:22.020053 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:36:22.019763    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4\": container with ID starting with 4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4 not found: ID does not exist" containerID="4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4"
Jul 28 12:36:22.020053 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019793    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4"} err="failed to get container status \"4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4\": rpc error: code = NotFound desc = could not find container \"4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4\": container with ID starting with 4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4 not found: ID does not exist"
Jul 28 12:36:22.020053 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.019816    2569 scope.go:117] "RemoveContainer" containerID="5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38"
Jul 28 12:36:22.020301 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.020272    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:36:22.020410 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:36:22.020384    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38\": container with ID starting with 5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38 not found: ID does not exist" containerID="5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38"
Jul 28 12:36:22.020475 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.020418    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38"} err="failed to get container status \"5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38\": rpc error: code = NotFound desc = could not find container \"5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38\": container with ID starting with 5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38 not found: ID does not exist"
Jul 28 12:36:22.020475 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.020443    2569 scope.go:117] "RemoveContainer" containerID="84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98"
Jul 28 12:36:22.020681 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.020654    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:22.020807 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.020785    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:22.021191 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:36:22.021049    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98\": container with ID starting with 84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98 not found: ID does not exist" containerID="84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98"
Jul 28 12:36:22.021191 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.021085    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98"} err="failed to get container status \"84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98\": rpc error: code = NotFound desc = could not find container \"84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98\": container with ID starting with 84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98 not found: ID does not exist"
Jul 28 12:36:22.021191 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.021107    2569 scope.go:117] "RemoveContainer" containerID="30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03"
Jul 28 12:36:22.021446 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:36:22.021422    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03\": container with ID starting with 30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03 not found: ID does not exist" containerID="30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03"
Jul 28 12:36:22.021511 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.021457    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03"} err="failed to get container status \"30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03\": rpc error: code = NotFound desc = could not find container \"30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03\": container with ID starting with 30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03 not found: ID does not exist"
Jul 28 12:36:22.021511 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.021478    2569 scope.go:117] "RemoveContainer" containerID="7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885"
Jul 28 12:36:22.021797 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:36:22.021769    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885\": container with ID starting with 7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885 not found: ID does not exist" containerID="7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885"
Jul 28 12:36:22.021873 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.021804    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885"} err="failed to get container status \"7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885\": rpc error: code = NotFound desc = could not find container \"7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885\": container with ID starting with 7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885 not found: ID does not exist"
Jul 28 12:36:22.021873 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.021824    2569 scope.go:117] "RemoveContainer" containerID="9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde"
Jul 28 12:36:22.022170 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.022131    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde"} err="failed to get container status \"9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde\": rpc error: code = NotFound desc = could not find container \"9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde\": container with ID starting with 9f0a768ea3937625c03d62defd3f21ed5108f36370bc55ca2a4e2934ef75efde not found: ID does not exist"
Jul 28 12:36:22.022259 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.022173    2569 scope.go:117] "RemoveContainer" containerID="703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747"
Jul 28 12:36:22.022482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.022456    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747"} err="failed to get container status \"703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747\": rpc error: code = NotFound desc = could not find container \"703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747\": container with ID starting with 703b23263cee2035f4e16e2c21fb9f253f53ca1bff7c69c2e757b99c2abd6747 not found: ID does not exist"
Jul 28 12:36:22.022565 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.022485    2569 scope.go:117] "RemoveContainer" containerID="4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4"
Jul 28 12:36:22.022748 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.022719    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4"} err="failed to get container status \"4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4\": rpc error: code = NotFound desc = could not find container \"4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4\": container with ID starting with 4fd997b0bc338333b716c7c0ecb09a2a2ecedd4a670d3148a300c02f49f64fd4 not found: ID does not exist"
Jul 28 12:36:22.022835 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.022751    2569 scope.go:117] "RemoveContainer" containerID="5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38"
Jul 28 12:36:22.022940 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.022919    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-kube-api-access-v2qhn" (OuterVolumeSpecName: "kube-api-access-v2qhn") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "kube-api-access-v2qhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:36:22.023065 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023028    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38"} err="failed to get container status \"5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38\": rpc error: code = NotFound desc = could not find container \"5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38\": container with ID starting with 5e719628b06eaef5e179cceeb9df3aebdcd72e33c9a44bdb30f8ab549201dd38 not found: ID does not exist"
Jul 28 12:36:22.023065 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023050    2569 scope.go:117] "RemoveContainer" containerID="84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98"
Jul 28 12:36:22.023252 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023231    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-out" (OuterVolumeSpecName: "config-out") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:36:22.023320 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023232    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:22.023414 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023389    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98"} err="failed to get container status \"84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98\": rpc error: code = NotFound desc = could not find container \"84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98\": container with ID starting with 84dfef2d1585635b0dcc102ece618dd0d0a667b3ff304584e5cbd8f82d6d9b98 not found: ID does not exist"
Jul 28 12:36:22.023477 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023420    2569 scope.go:117] "RemoveContainer" containerID="30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03"
Jul 28 12:36:22.023538 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023518    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:36:22.023714 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023688    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03"} err="failed to get container status \"30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03\": rpc error: code = NotFound desc = could not find container \"30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03\": container with ID starting with 30d2c9bfd7375424f0fcdc381b7f086e89c0f5f1a9ce3c6a9805e5858173dc03 not found: ID does not exist"
Jul 28 12:36:22.023806 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023717    2569 scope.go:117] "RemoveContainer" containerID="7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885"
Jul 28 12:36:22.023948 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023924    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-volume" (OuterVolumeSpecName: "config-volume") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:22.024062 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.023962    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885"} err="failed to get container status \"7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885\": rpc error: code = NotFound desc = could not find container \"7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885\": container with ID starting with 7f25b5f6f7540c8acd016769a5246ff0747c8a6dac97a5fafabfacd7308eb885 not found: ID does not exist"
Jul 28 12:36:22.024514 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.024487    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:22.024707 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.024669    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:22.024775 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.024726    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:22.027366 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.027345    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:22.033710 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.033689    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-web-config" (OuterVolumeSpecName: "web-config") pod "d1579f16-b3d2-4977-a9d9-b1804f6089dd" (UID: "d1579f16-b3d2-4977-a9d9-b1804f6089dd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:22.121416 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121377    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2qhn\" (UniqueName: \"kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-kube-api-access-v2qhn\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121416 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121404    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121416 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121414    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-web-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121416 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121425    2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-cluster-tls-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121434    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-metrics-client-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121445    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-main-db\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121455    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-main-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121463    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1579f16-b3d2-4977-a9d9-b1804f6089dd-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121471    2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-volume\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121481    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121492    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1579f16-b3d2-4977-a9d9-b1804f6089dd-config-out\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121502    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1579f16-b3d2-4977-a9d9-b1804f6089dd-tls-assets\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.121738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.121510    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1579f16-b3d2-4977-a9d9-b1804f6089dd-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:36:22.290362 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.290330    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:22.295003 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.294962    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:22.320992 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.320969    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:22.321263 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321250    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77fac7f3-5fa0-4364-afd1-c8739b27d163" containerName="console"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321264    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="77fac7f3-5fa0-4364-afd1-c8739b27d163" containerName="console"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321273    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68d448f-212b-4487-85c8-a628794a75d3" containerName="console"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321279    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68d448f-212b-4487-85c8-a628794a75d3" containerName="console"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321286    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="prom-label-proxy"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321294    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="prom-label-proxy"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321301    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy-metric"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321306    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy-metric"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321316    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="config-reloader"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321321    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="config-reloader"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321327    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy"
Jul 28 12:36:22.321334 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321333    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321340    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="alertmanager"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321346    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="alertmanager"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321351    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy-web"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321357    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy-web"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321369    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="init-config-reloader"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321375    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="init-config-reloader"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321413    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321419    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="prom-label-proxy"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321426    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="alertmanager"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321431    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy-web"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321438    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="config-reloader"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321443    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68d448f-212b-4487-85c8-a628794a75d3" containerName="console"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321447    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" containerName="kube-rbac-proxy-metric"
Jul 28 12:36:22.321675 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.321453    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="77fac7f3-5fa0-4364-afd1-c8739b27d163" containerName="console"
Jul 28 12:36:22.325044 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.325026    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.330034 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.329362    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 12:36:22.330034 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.329629    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 12:36:22.330034 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.329794    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 12:36:22.330034 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.329938    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 12:36:22.330282 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.330261    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 12:36:22.330282 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.330268    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 12:36:22.330386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.330288    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 12:36:22.330386 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.330304    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 12:36:22.330612 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.330595    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6247m\""
Jul 28 12:36:22.336489 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.336462    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 12:36:22.341301 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.341268    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:22.423315 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423323    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7465f793-3b4e-43d5-92c9-a538ed297a7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7465f793-3b4e-43d5-92c9-a538ed297a7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423398    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7465f793-3b4e-43d5-92c9-a538ed297a7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423624 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423478    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7465f793-3b4e-43d5-92c9-a538ed297a7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423624 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423518    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423624 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423624 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423578    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v64xj\" (UniqueName: \"kubernetes.io/projected/7465f793-3b4e-43d5-92c9-a538ed297a7b-kube-api-access-v64xj\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423624 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423787 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7465f793-3b4e-43d5-92c9-a538ed297a7b-config-out\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423787 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423663    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-web-config\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.423787 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.423678    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524221 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524144    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7465f793-3b4e-43d5-92c9-a538ed297a7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524221 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7465f793-3b4e-43d5-92c9-a538ed297a7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524221 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524195    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7465f793-3b4e-43d5-92c9-a538ed297a7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524221 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524220    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524493 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524366    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7465f793-3b4e-43d5-92c9-a538ed297a7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524493 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524422    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524595 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524494    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524595 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524521    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v64xj\" (UniqueName: \"kubernetes.io/projected/7465f793-3b4e-43d5-92c9-a538ed297a7b-kube-api-access-v64xj\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524595 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524560    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524748 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7465f793-3b4e-43d5-92c9-a538ed297a7b-config-out\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524748 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-web-config\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524748 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.524748 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524669    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.525014 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524960    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7465f793-3b4e-43d5-92c9-a538ed297a7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.525099 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.524560    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7465f793-3b4e-43d5-92c9-a538ed297a7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.525267 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.525242    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7465f793-3b4e-43d5-92c9-a538ed297a7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.527400 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.527368    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7465f793-3b4e-43d5-92c9-a538ed297a7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.527501 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.527381    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.527501 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.527385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.527501 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.527471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.527663 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.527471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.527663 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.527560    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-web-config\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.527778 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.527760    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.528078 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.528061    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7465f793-3b4e-43d5-92c9-a538ed297a7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.529037 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.529023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7465f793-3b4e-43d5-92c9-a538ed297a7b-config-out\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.533613 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.533590    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v64xj\" (UniqueName: \"kubernetes.io/projected/7465f793-3b4e-43d5-92c9-a538ed297a7b-kube-api-access-v64xj\") pod \"alertmanager-main-0\" (UID: \"7465f793-3b4e-43d5-92c9-a538ed297a7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.637330 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.637299    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:22.767500 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.767469    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:22.770452 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:36:22.770426    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7465f793_3b4e_43d5_92c9_a538ed297a7b.slice/crio-5c984938c00b8a1d8003c59f35d9aaced7555b511766fe659063492f236b4209 WatchSource:0}: Error finding container 5c984938c00b8a1d8003c59f35d9aaced7555b511766fe659063492f236b4209: Status 404 returned error can't find the container with id 5c984938c00b8a1d8003c59f35d9aaced7555b511766fe659063492f236b4209
Jul 28 12:36:22.977648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.977610    2569 generic.go:356] "Generic (PLEG): container finished" podID="7465f793-3b4e-43d5-92c9-a538ed297a7b" containerID="ba33614a2df2e6745e8ca1d5ce7556aef31fe2c990898ef6d0ef683b7ed9028d" exitCode=0
Jul 28 12:36:22.978021 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.977681    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7465f793-3b4e-43d5-92c9-a538ed297a7b","Type":"ContainerDied","Data":"ba33614a2df2e6745e8ca1d5ce7556aef31fe2c990898ef6d0ef683b7ed9028d"}
Jul 28 12:36:22.978021 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:22.977744    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7465f793-3b4e-43d5-92c9-a538ed297a7b","Type":"ContainerStarted","Data":"5c984938c00b8a1d8003c59f35d9aaced7555b511766fe659063492f236b4209"}
Jul 28 12:36:23.985942 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:23.985852    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7465f793-3b4e-43d5-92c9-a538ed297a7b","Type":"ContainerStarted","Data":"30e0f9d7d000ba06c601df5c92c8992af2f762cdacb7a4a601c8c95babd65ce2"}
Jul 28 12:36:23.985942 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:23.985929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7465f793-3b4e-43d5-92c9-a538ed297a7b","Type":"ContainerStarted","Data":"bdebd061007e8ec96b386d88c81f41f10269144c203cd69c79fc3de55a6929b0"}
Jul 28 12:36:23.986449 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:23.985968    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7465f793-3b4e-43d5-92c9-a538ed297a7b","Type":"ContainerStarted","Data":"5be174747ad079fe35e173554e312afd0097031e249e4320c9251f53f3d50678"}
Jul 28 12:36:23.986449 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:23.986033    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7465f793-3b4e-43d5-92c9-a538ed297a7b","Type":"ContainerStarted","Data":"fd9b7ab0a623ecd3b84f8766ba25e19a5589c9fa934ab9fc2cd942b668ece72e"}
Jul 28 12:36:23.986449 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:23.986076    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7465f793-3b4e-43d5-92c9-a538ed297a7b","Type":"ContainerStarted","Data":"2789b381d3dfe5c5383e293f73dcb6a348db544aa77951883e44f1fc10bf9792"}
Jul 28 12:36:23.986449 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:23.986118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7465f793-3b4e-43d5-92c9-a538ed297a7b","Type":"ContainerStarted","Data":"def857d80836158ebccf6e047a121560aff95edf093d4c07ead3f0967e636870"}
Jul 28 12:36:24.011611 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.011545    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.9941471339999999 podStartE2EDuration="2.01153052s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.383386ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:24.010300679 +0000 UTC m=+152.225817343" watchObservedRunningTime="2026-07-28 12:36:24.01153052 +0000 UTC m=+152.227047171"
Jul 28 12:36:24.255501 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.255420    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1579f16-b3d2-4977-a9d9-b1804f6089dd" path="/var/lib/kubelet/pods/d1579f16-b3d2-4977-a9d9-b1804f6089dd/volumes"
Jul 28 12:36:24.720975 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.720942    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-765d778646-7ws2w"]
Jul 28 12:36:24.723755 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.723730    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.726494 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.726457    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 12:36:24.726742 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.726718    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tgkdm\""
Jul 28 12:36:24.726880 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.726718    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 12:36:24.726880 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.726791    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 12:36:24.727005 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.726905    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 12:36:24.727412 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.727392    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 12:36:24.732177 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.732149    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 12:36:24.736665 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.736643    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-765d778646-7ws2w"]
Jul 28 12:36:24.745394 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.745358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.745528 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.745407    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7742da-776d-44e7-8004-83c4d730e8cf-serving-certs-ca-bundle\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.745528 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.745441    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-telemeter-client-tls\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.745528 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.745474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4r6t\" (UniqueName: \"kubernetes.io/projected/7b7742da-776d-44e7-8004-83c4d730e8cf-kube-api-access-f4r6t\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.745528 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.745521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7742da-776d-44e7-8004-83c4d730e8cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.745719 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.745572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-secret-telemeter-client\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.745719 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.745596    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-federate-client-tls\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.745719 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.745632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b7742da-776d-44e7-8004-83c4d730e8cf-metrics-client-ca\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.847008 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.846949    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b7742da-776d-44e7-8004-83c4d730e8cf-metrics-client-ca\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.847204 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.847028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.847204 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.847051    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7742da-776d-44e7-8004-83c4d730e8cf-serving-certs-ca-bundle\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.847204 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.847072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-telemeter-client-tls\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.847204 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.847098    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4r6t\" (UniqueName: \"kubernetes.io/projected/7b7742da-776d-44e7-8004-83c4d730e8cf-kube-api-access-f4r6t\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.847204 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.847145    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7742da-776d-44e7-8004-83c4d730e8cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.847204 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.847203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-secret-telemeter-client\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.847540 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.847227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-federate-client-tls\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.847865 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.847815    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b7742da-776d-44e7-8004-83c4d730e8cf-metrics-client-ca\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.847865 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.847841    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7742da-776d-44e7-8004-83c4d730e8cf-serving-certs-ca-bundle\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.848071 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.847959    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7742da-776d-44e7-8004-83c4d730e8cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.850102 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.850074    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-telemeter-client-tls\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.850211 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.850125    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-secret-telemeter-client\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.852076 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.850579    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-federate-client-tls\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.852076 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.850941    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b7742da-776d-44e7-8004-83c4d730e8cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:24.855285 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:24.855265    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4r6t\" (UniqueName: \"kubernetes.io/projected/7b7742da-776d-44e7-8004-83c4d730e8cf-kube-api-access-f4r6t\") pod \"telemeter-client-765d778646-7ws2w\" (UID: \"7b7742da-776d-44e7-8004-83c4d730e8cf\") " pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:25.037009 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:25.036914    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-765d778646-7ws2w"
Jul 28 12:36:25.185381 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:25.185343    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-765d778646-7ws2w"]
Jul 28 12:36:25.191608 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:36:25.191577    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7742da_776d_44e7_8004_83c4d730e8cf.slice/crio-15b7dcea824570f9727a889ee0f1160f80d38b4a18d157975362fdedecd40e32 WatchSource:0}: Error finding container 15b7dcea824570f9727a889ee0f1160f80d38b4a18d157975362fdedecd40e32: Status 404 returned error can't find the container with id 15b7dcea824570f9727a889ee0f1160f80d38b4a18d157975362fdedecd40e32
Jul 28 12:36:25.996945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:25.996906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-765d778646-7ws2w" event={"ID":"7b7742da-776d-44e7-8004-83c4d730e8cf","Type":"ContainerStarted","Data":"15b7dcea824570f9727a889ee0f1160f80d38b4a18d157975362fdedecd40e32"}
Jul 28 12:36:28.008090 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:28.008052    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-765d778646-7ws2w" event={"ID":"7b7742da-776d-44e7-8004-83c4d730e8cf","Type":"ContainerStarted","Data":"9341d62500df727084cf25dda36d2355e36fc87595dca13c4f3efc2e0ae624b8"}
Jul 28 12:36:28.008553 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:28.008114    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-765d778646-7ws2w" event={"ID":"7b7742da-776d-44e7-8004-83c4d730e8cf","Type":"ContainerStarted","Data":"c12f9f331095441f362c8a4a8797252b3adc38effe69c2c1ddb0dee66c4d57d3"}
Jul 28 12:36:28.008553 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:28.008142    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-765d778646-7ws2w" event={"ID":"7b7742da-776d-44e7-8004-83c4d730e8cf","Type":"ContainerStarted","Data":"d9e6cc4d63c5359f15788c032b6e95669a89538272ddbf8cd17e5b7bbd2b62ca"}
Jul 28 12:36:28.029560 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:28.029510    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-765d778646-7ws2w" podStartSLOduration=2.137899207 podStartE2EDuration="4.029495446s" totalImagesPullingTime="1.891596239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:28.027769971 +0000 UTC m=+156.243286622" watchObservedRunningTime="2026-07-28 12:36:28.029495446 +0000 UTC m=+156.245012097"
Jul 28 12:36:29.347321 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.347288    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5589f56475-rzhhd"]
Jul 28 12:36:29.349540 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.349522    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.361807 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.361783    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5589f56475-rzhhd"]
Jul 28 12:36:29.392251 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.392229    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ks8f\" (UniqueName: \"kubernetes.io/projected/5403f665-e9e7-46de-98c6-3c87500400aa-kube-api-access-7ks8f\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.392360 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.392263    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-oauth-config\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.392360 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.392284    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-console-config\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.392429 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.392376    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-service-ca\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.392429 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.392404    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-trusted-ca-bundle\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.392494 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.392466    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-serving-cert\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.392527 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.392494    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-oauth-serving-cert\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.493355 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.493316    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ks8f\" (UniqueName: \"kubernetes.io/projected/5403f665-e9e7-46de-98c6-3c87500400aa-kube-api-access-7ks8f\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.493355 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.493355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-oauth-config\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.493583 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.493375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-console-config\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.493583 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.493420    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-service-ca\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.493583 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.493436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-trusted-ca-bundle\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.493583 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.493462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-serving-cert\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.493583 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.493488    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-oauth-serving-cert\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.494295 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.494270    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-service-ca\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.494431 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.494277    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-console-config\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.494431 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.494277    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-oauth-serving-cert\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.494431 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.494340    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-trusted-ca-bundle\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.495823 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.495794    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-oauth-config\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.495972 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.495957    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-serving-cert\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.500763 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.500743    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ks8f\" (UniqueName: \"kubernetes.io/projected/5403f665-e9e7-46de-98c6-3c87500400aa-kube-api-access-7ks8f\") pod \"console-5589f56475-rzhhd\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") " pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.658777 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.658681    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:29.785565 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:29.785539    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5589f56475-rzhhd"]
Jul 28 12:36:29.788318 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:36:29.788287    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5403f665_e9e7_46de_98c6_3c87500400aa.slice/crio-eb2ce67df32e8c1eeb5f0a87458a76fa9a8e56721a844dbf6564c3293c543e10 WatchSource:0}: Error finding container eb2ce67df32e8c1eeb5f0a87458a76fa9a8e56721a844dbf6564c3293c543e10: Status 404 returned error can't find the container with id eb2ce67df32e8c1eeb5f0a87458a76fa9a8e56721a844dbf6564c3293c543e10
Jul 28 12:36:30.020058 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:30.020018    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5589f56475-rzhhd" event={"ID":"5403f665-e9e7-46de-98c6-3c87500400aa","Type":"ContainerStarted","Data":"db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c"}
Jul 28 12:36:30.020228 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:30.020097    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5589f56475-rzhhd" event={"ID":"5403f665-e9e7-46de-98c6-3c87500400aa","Type":"ContainerStarted","Data":"eb2ce67df32e8c1eeb5f0a87458a76fa9a8e56721a844dbf6564c3293c543e10"}
Jul 28 12:36:30.037572 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:30.037526    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5589f56475-rzhhd" podStartSLOduration=1.037412827 podStartE2EDuration="1.037412827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:30.037112986 +0000 UTC m=+158.252629636" watchObservedRunningTime="2026-07-28 12:36:30.037412827 +0000 UTC m=+158.252929480"
Jul 28 12:36:39.659739 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:39.659703    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:39.660206 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:39.659765    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:39.664543 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:39.664514    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:40.063180 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:40.063154    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:36:40.110323 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:36:40.110294    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-db8f99589-tqpzl"]
Jul 28 12:37:05.129574 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.129527    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-db8f99589-tqpzl" podUID="26e89fc8-8390-4a8f-b69e-6b7c948ec15c" containerName="console" containerID="cri-o://8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5" gracePeriod=15
Jul 28 12:37:05.367733 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.367709    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:37:05.392546 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.392491    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-config\") pod \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") "
Jul 28 12:37:05.392546 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.392541    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-oauth-config\") pod \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") "
Jul 28 12:37:05.392682 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.392564    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-service-ca\") pod \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") "
Jul 28 12:37:05.392682 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.392590    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfm2m\" (UniqueName: \"kubernetes.io/projected/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-kube-api-access-vfm2m\") pod \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") "
Jul 28 12:37:05.392682 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.392608    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-oauth-serving-cert\") pod \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") "
Jul 28 12:37:05.392682 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.392627    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-trusted-ca-bundle\") pod \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") "
Jul 28 12:37:05.392682 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.392665    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-serving-cert\") pod \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\" (UID: \"26e89fc8-8390-4a8f-b69e-6b7c948ec15c\") "
Jul 28 12:37:05.393078 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.392912    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-config" (OuterVolumeSpecName: "console-config") pod "26e89fc8-8390-4a8f-b69e-6b7c948ec15c" (UID: "26e89fc8-8390-4a8f-b69e-6b7c948ec15c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:05.393078 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.393062    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "26e89fc8-8390-4a8f-b69e-6b7c948ec15c" (UID: "26e89fc8-8390-4a8f-b69e-6b7c948ec15c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:05.393231 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.393095    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-service-ca" (OuterVolumeSpecName: "service-ca") pod "26e89fc8-8390-4a8f-b69e-6b7c948ec15c" (UID: "26e89fc8-8390-4a8f-b69e-6b7c948ec15c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:05.393231 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.393158    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "26e89fc8-8390-4a8f-b69e-6b7c948ec15c" (UID: "26e89fc8-8390-4a8f-b69e-6b7c948ec15c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:05.394710 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.394687    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "26e89fc8-8390-4a8f-b69e-6b7c948ec15c" (UID: "26e89fc8-8390-4a8f-b69e-6b7c948ec15c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:05.394808 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.394690    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-kube-api-access-vfm2m" (OuterVolumeSpecName: "kube-api-access-vfm2m") pod "26e89fc8-8390-4a8f-b69e-6b7c948ec15c" (UID: "26e89fc8-8390-4a8f-b69e-6b7c948ec15c"). InnerVolumeSpecName "kube-api-access-vfm2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:05.394808 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.394798    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "26e89fc8-8390-4a8f-b69e-6b7c948ec15c" (UID: "26e89fc8-8390-4a8f-b69e-6b7c948ec15c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:05.494189 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.494150    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:37:05.494189 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.494182    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-oauth-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:37:05.494189 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.494193    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:37:05.494433 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.494202    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfm2m\" (UniqueName: \"kubernetes.io/projected/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-kube-api-access-vfm2m\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:37:05.494433 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.494214    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-oauth-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:37:05.494433 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.494223    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-trusted-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:37:05.494433 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:05.494231    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26e89fc8-8390-4a8f-b69e-6b7c948ec15c-console-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:37:06.162029 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:06.161978    2569 generic.go:356] "Generic (PLEG): container finished" podID="26e89fc8-8390-4a8f-b69e-6b7c948ec15c" containerID="8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5" exitCode=0
Jul 28 12:37:06.162400 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:06.162041    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db8f99589-tqpzl"
Jul 28 12:37:06.162400 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:06.162060    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db8f99589-tqpzl" event={"ID":"26e89fc8-8390-4a8f-b69e-6b7c948ec15c","Type":"ContainerDied","Data":"8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5"}
Jul 28 12:37:06.162400 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:06.162111    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db8f99589-tqpzl" event={"ID":"26e89fc8-8390-4a8f-b69e-6b7c948ec15c","Type":"ContainerDied","Data":"a31097886fddb12d0405ebf90d60b2772c3b6857bd3e27eb1a68e1911dc74de3"}
Jul 28 12:37:06.162400 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:06.162130    2569 scope.go:117] "RemoveContainer" containerID="8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5"
Jul 28 12:37:06.170634 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:06.170620    2569 scope.go:117] "RemoveContainer" containerID="8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5"
Jul 28 12:37:06.170899 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:37:06.170878    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5\": container with ID starting with 8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5 not found: ID does not exist" containerID="8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5"
Jul 28 12:37:06.170946 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:06.170908    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5"} err="failed to get container status \"8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5\": rpc error: code = NotFound desc = could not find container \"8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5\": container with ID starting with 8554ab714492a07bf8d013db31778cfc43b261c26e513040d14520f90e3974f5 not found: ID does not exist"
Jul 28 12:37:06.182287 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:06.182268    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-db8f99589-tqpzl"]
Jul 28 12:37:06.187710 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:06.187681    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-db8f99589-tqpzl"]
Jul 28 12:37:06.250425 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:06.250387    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e89fc8-8390-4a8f-b69e-6b7c948ec15c" path="/var/lib/kubelet/pods/26e89fc8-8390-4a8f-b69e-6b7c948ec15c/volumes"
Jul 28 12:37:46.122515 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.122482    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8499c78db4-bbr8h"]
Jul 28 12:37:46.122957 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.122797    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e89fc8-8390-4a8f-b69e-6b7c948ec15c" containerName="console"
Jul 28 12:37:46.122957 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.122808    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e89fc8-8390-4a8f-b69e-6b7c948ec15c" containerName="console"
Jul 28 12:37:46.122957 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.122882    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e89fc8-8390-4a8f-b69e-6b7c948ec15c" containerName="console"
Jul 28 12:37:46.125631 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.125615    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.138508 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.138486    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8499c78db4-bbr8h"]
Jul 28 12:37:46.156483 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.156457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-config\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.156625 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.156490    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7wdv\" (UniqueName: \"kubernetes.io/projected/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-kube-api-access-p7wdv\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.156625 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.156527    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-service-ca\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.156625 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.156549    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-oauth-serving-cert\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.156625 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.156612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-oauth-config\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.156762 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.156652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-trusted-ca-bundle\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.156762 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.156722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-serving-cert\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.257761 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.257734    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-trusted-ca-bundle\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.257888 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.257778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-serving-cert\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.257888 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.257812    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-config\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.257888 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.257827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7wdv\" (UniqueName: \"kubernetes.io/projected/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-kube-api-access-p7wdv\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.257888 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.257855    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-service-ca\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.257888 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.257876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-oauth-serving-cert\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.258242 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.257914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-oauth-config\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.258608 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.258571    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-config\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.258723 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.258634    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-service-ca\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.258723 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.258666    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-trusted-ca-bundle\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.258723 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.258683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-oauth-serving-cert\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.260404 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.260373    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-serving-cert\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.260404 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.260384    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-oauth-config\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.265320 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.265301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7wdv\" (UniqueName: \"kubernetes.io/projected/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-kube-api-access-p7wdv\") pod \"console-8499c78db4-bbr8h\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") " pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.435035 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.434925    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:46.569514 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:46.569353    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8499c78db4-bbr8h"]
Jul 28 12:37:46.572075 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:37:46.572046    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78a9ce1_a9d9_4ec5_9500_e302915e34eb.slice/crio-1f9beb8d8e53f75b808b838897dafae344591c494fd4af2aa451038617363285 WatchSource:0}: Error finding container 1f9beb8d8e53f75b808b838897dafae344591c494fd4af2aa451038617363285: Status 404 returned error can't find the container with id 1f9beb8d8e53f75b808b838897dafae344591c494fd4af2aa451038617363285
Jul 28 12:37:47.317362 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:47.317326    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8499c78db4-bbr8h" event={"ID":"c78a9ce1-a9d9-4ec5-9500-e302915e34eb","Type":"ContainerStarted","Data":"6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486"}
Jul 28 12:37:47.317727 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:47.317398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8499c78db4-bbr8h" event={"ID":"c78a9ce1-a9d9-4ec5-9500-e302915e34eb","Type":"ContainerStarted","Data":"1f9beb8d8e53f75b808b838897dafae344591c494fd4af2aa451038617363285"}
Jul 28 12:37:47.335525 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:47.335474    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8499c78db4-bbr8h" podStartSLOduration=1.335457684 podStartE2EDuration="1.335457684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:37:47.3337694 +0000 UTC m=+235.549286052" watchObservedRunningTime="2026-07-28 12:37:47.335457684 +0000 UTC m=+235.550974335"
Jul 28 12:37:56.435242 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:56.435207    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:56.435644 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:56.435259    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:56.439458 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:56.439439    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:57.359114 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:57.359081    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:37:57.443736 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:37:57.443706    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5589f56475-rzhhd"]
Jul 28 12:38:22.462509 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.462401    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5589f56475-rzhhd" podUID="5403f665-e9e7-46de-98c6-3c87500400aa" containerName="console" containerID="cri-o://db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c" gracePeriod=15
Jul 28 12:38:22.705310 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.705277    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:38:22.868307 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.868209    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-serving-cert\") pod \"5403f665-e9e7-46de-98c6-3c87500400aa\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") "
Jul 28 12:38:22.868307 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.868274    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-oauth-serving-cert\") pod \"5403f665-e9e7-46de-98c6-3c87500400aa\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") "
Jul 28 12:38:22.868538 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.868320    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-trusted-ca-bundle\") pod \"5403f665-e9e7-46de-98c6-3c87500400aa\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") "
Jul 28 12:38:22.868538 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.868351    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-console-config\") pod \"5403f665-e9e7-46de-98c6-3c87500400aa\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") "
Jul 28 12:38:22.868538 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.868408    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ks8f\" (UniqueName: \"kubernetes.io/projected/5403f665-e9e7-46de-98c6-3c87500400aa-kube-api-access-7ks8f\") pod \"5403f665-e9e7-46de-98c6-3c87500400aa\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") "
Jul 28 12:38:22.868538 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.868436    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-oauth-config\") pod \"5403f665-e9e7-46de-98c6-3c87500400aa\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") "
Jul 28 12:38:22.868538 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.868468    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-service-ca\") pod \"5403f665-e9e7-46de-98c6-3c87500400aa\" (UID: \"5403f665-e9e7-46de-98c6-3c87500400aa\") "
Jul 28 12:38:22.868838 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.868743    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5403f665-e9e7-46de-98c6-3c87500400aa" (UID: "5403f665-e9e7-46de-98c6-3c87500400aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:22.868838 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.868763    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5403f665-e9e7-46de-98c6-3c87500400aa" (UID: "5403f665-e9e7-46de-98c6-3c87500400aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:22.868950 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.868835    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-console-config" (OuterVolumeSpecName: "console-config") pod "5403f665-e9e7-46de-98c6-3c87500400aa" (UID: "5403f665-e9e7-46de-98c6-3c87500400aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:22.869076 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.869056    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-service-ca" (OuterVolumeSpecName: "service-ca") pod "5403f665-e9e7-46de-98c6-3c87500400aa" (UID: "5403f665-e9e7-46de-98c6-3c87500400aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:22.870484 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.870456    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5403f665-e9e7-46de-98c6-3c87500400aa-kube-api-access-7ks8f" (OuterVolumeSpecName: "kube-api-access-7ks8f") pod "5403f665-e9e7-46de-98c6-3c87500400aa" (UID: "5403f665-e9e7-46de-98c6-3c87500400aa"). InnerVolumeSpecName "kube-api-access-7ks8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:38:22.870484 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.870475    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5403f665-e9e7-46de-98c6-3c87500400aa" (UID: "5403f665-e9e7-46de-98c6-3c87500400aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:22.870625 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.870534    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5403f665-e9e7-46de-98c6-3c87500400aa" (UID: "5403f665-e9e7-46de-98c6-3c87500400aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:22.969743 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.969702    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ks8f\" (UniqueName: \"kubernetes.io/projected/5403f665-e9e7-46de-98c6-3c87500400aa-kube-api-access-7ks8f\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:38:22.969743 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.969735    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-oauth-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:38:22.969743 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.969747    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:38:22.969983 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.969761    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5403f665-e9e7-46de-98c6-3c87500400aa-console-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:38:22.969983 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.969774    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-oauth-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:38:22.969983 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.969785    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-trusted-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:38:22.969983 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:22.969793    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5403f665-e9e7-46de-98c6-3c87500400aa-console-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:38:23.451705 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:23.451668    2569 generic.go:356] "Generic (PLEG): container finished" podID="5403f665-e9e7-46de-98c6-3c87500400aa" containerID="db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c" exitCode=0
Jul 28 12:38:23.451907 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:23.451715    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5589f56475-rzhhd"
Jul 28 12:38:23.451907 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:23.451756    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5589f56475-rzhhd" event={"ID":"5403f665-e9e7-46de-98c6-3c87500400aa","Type":"ContainerDied","Data":"db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c"}
Jul 28 12:38:23.451907 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:23.451810    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5589f56475-rzhhd" event={"ID":"5403f665-e9e7-46de-98c6-3c87500400aa","Type":"ContainerDied","Data":"eb2ce67df32e8c1eeb5f0a87458a76fa9a8e56721a844dbf6564c3293c543e10"}
Jul 28 12:38:23.451907 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:23.451830    2569 scope.go:117] "RemoveContainer" containerID="db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c"
Jul 28 12:38:23.460975 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:23.460950    2569 scope.go:117] "RemoveContainer" containerID="db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c"
Jul 28 12:38:23.461243 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:38:23.461227    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c\": container with ID starting with db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c not found: ID does not exist" containerID="db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c"
Jul 28 12:38:23.461290 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:23.461252    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c"} err="failed to get container status \"db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c\": rpc error: code = NotFound desc = could not find container \"db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c\": container with ID starting with db16482c1be08fc75c739082b2c644209ffd8bf866474f917ff4d3a06578df2c not found: ID does not exist"
Jul 28 12:38:23.473399 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:23.473373    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5589f56475-rzhhd"]
Jul 28 12:38:23.481574 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:23.477930    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5589f56475-rzhhd"]
Jul 28 12:38:24.250905 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:24.250871    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5403f665-e9e7-46de-98c6-3c87500400aa" path="/var/lib/kubelet/pods/5403f665-e9e7-46de-98c6-3c87500400aa/volumes"
Jul 28 12:38:52.103325 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:38:52.103297    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 12:40:55.790649 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.790622    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"]
Jul 28 12:40:55.791097 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.790917    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5403f665-e9e7-46de-98c6-3c87500400aa" containerName="console"
Jul 28 12:40:55.791097 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.790928    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5403f665-e9e7-46de-98c6-3c87500400aa" containerName="console"
Jul 28 12:40:55.791097 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.790980    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5403f665-e9e7-46de-98c6-3c87500400aa" containerName="console"
Jul 28 12:40:55.794036 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.794017    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:55.798372 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.798351    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tpvr2\""
Jul 28 12:40:55.799005 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.798968    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 12:40:55.799100 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.799020    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 12:40:55.840834 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.840808    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"]
Jul 28 12:40:55.936705 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.936667    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:55.936871 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.936712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4dt\" (UniqueName: \"kubernetes.io/projected/b927db3f-ec6c-49f6-86be-07dba2864b35-kube-api-access-xs4dt\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:55.936871 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:55.936802    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:56.038022 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:56.037979    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:56.038188 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:56.038029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4dt\" (UniqueName: \"kubernetes.io/projected/b927db3f-ec6c-49f6-86be-07dba2864b35-kube-api-access-xs4dt\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:56.038188 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:56.038088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:56.038392 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:56.038373    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:56.038429 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:56.038393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:56.054200 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:56.054143    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4dt\" (UniqueName: \"kubernetes.io/projected/b927db3f-ec6c-49f6-86be-07dba2864b35-kube-api-access-xs4dt\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:56.103271 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:56.103246    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:40:56.237217 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:56.237194    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"]
Jul 28 12:40:56.239812 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:40:56.239783    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb927db3f_ec6c_49f6_86be_07dba2864b35.slice/crio-ce129f17f306f446acc5a55e3211ccd8ef72181dfa098d00fee3bee80a064160 WatchSource:0}: Error finding container ce129f17f306f446acc5a55e3211ccd8ef72181dfa098d00fee3bee80a064160: Status 404 returned error can't find the container with id ce129f17f306f446acc5a55e3211ccd8ef72181dfa098d00fee3bee80a064160
Jul 28 12:40:56.241633 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:56.241616    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:40:57.023022 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:40:57.022967    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw" event={"ID":"b927db3f-ec6c-49f6-86be-07dba2864b35","Type":"ContainerStarted","Data":"ce129f17f306f446acc5a55e3211ccd8ef72181dfa098d00fee3bee80a064160"}
Jul 28 12:41:03.050580 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:03.050541    2569 generic.go:356] "Generic (PLEG): container finished" podID="b927db3f-ec6c-49f6-86be-07dba2864b35" containerID="89ca3ac884230be0f1322498aba9eac991cc65ed36eac703b053030d623023d3" exitCode=0
Jul 28 12:41:03.050960 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:03.050609    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw" event={"ID":"b927db3f-ec6c-49f6-86be-07dba2864b35","Type":"ContainerDied","Data":"89ca3ac884230be0f1322498aba9eac991cc65ed36eac703b053030d623023d3"}
Jul 28 12:41:06.065555 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:06.065521    2569 generic.go:356] "Generic (PLEG): container finished" podID="b927db3f-ec6c-49f6-86be-07dba2864b35" containerID="d8451189fd565be1ec1ebb2724aae7b0af488f81a150106bf6344515e60050bd" exitCode=0
Jul 28 12:41:06.065950 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:06.065608    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw" event={"ID":"b927db3f-ec6c-49f6-86be-07dba2864b35","Type":"ContainerDied","Data":"d8451189fd565be1ec1ebb2724aae7b0af488f81a150106bf6344515e60050bd"}
Jul 28 12:41:13.097002 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:13.096959    2569 generic.go:356] "Generic (PLEG): container finished" podID="b927db3f-ec6c-49f6-86be-07dba2864b35" containerID="7714db8f212e63e83cb0ef7686073cd8fe51500d3857f095df79b4831d8ae8b4" exitCode=0
Jul 28 12:41:13.097411 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:13.097021    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw" event={"ID":"b927db3f-ec6c-49f6-86be-07dba2864b35","Type":"ContainerDied","Data":"7714db8f212e63e83cb0ef7686073cd8fe51500d3857f095df79b4831d8ae8b4"}
Jul 28 12:41:14.230647 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:14.230621    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:41:14.293625 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:14.293598    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-bundle\") pod \"b927db3f-ec6c-49f6-86be-07dba2864b35\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") "
Jul 28 12:41:14.293805 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:14.293638    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-util\") pod \"b927db3f-ec6c-49f6-86be-07dba2864b35\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") "
Jul 28 12:41:14.293805 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:14.293667    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs4dt\" (UniqueName: \"kubernetes.io/projected/b927db3f-ec6c-49f6-86be-07dba2864b35-kube-api-access-xs4dt\") pod \"b927db3f-ec6c-49f6-86be-07dba2864b35\" (UID: \"b927db3f-ec6c-49f6-86be-07dba2864b35\") "
Jul 28 12:41:14.294302 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:14.294276    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-bundle" (OuterVolumeSpecName: "bundle") pod "b927db3f-ec6c-49f6-86be-07dba2864b35" (UID: "b927db3f-ec6c-49f6-86be-07dba2864b35"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:41:14.295826 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:14.295802    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b927db3f-ec6c-49f6-86be-07dba2864b35-kube-api-access-xs4dt" (OuterVolumeSpecName: "kube-api-access-xs4dt") pod "b927db3f-ec6c-49f6-86be-07dba2864b35" (UID: "b927db3f-ec6c-49f6-86be-07dba2864b35"). InnerVolumeSpecName "kube-api-access-xs4dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:41:14.297862 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:14.297834    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-util" (OuterVolumeSpecName: "util") pod "b927db3f-ec6c-49f6-86be-07dba2864b35" (UID: "b927db3f-ec6c-49f6-86be-07dba2864b35"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:41:14.394581 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:14.394485    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs4dt\" (UniqueName: \"kubernetes.io/projected/b927db3f-ec6c-49f6-86be-07dba2864b35-kube-api-access-xs4dt\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:41:14.394581 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:14.394531    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:41:14.394581 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:14.394541    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b927db3f-ec6c-49f6-86be-07dba2864b35-util\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:41:15.106272 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:15.106235    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw" event={"ID":"b927db3f-ec6c-49f6-86be-07dba2864b35","Type":"ContainerDied","Data":"ce129f17f306f446acc5a55e3211ccd8ef72181dfa098d00fee3bee80a064160"}
Jul 28 12:41:15.106464 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:15.106279    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55tbnw"
Jul 28 12:41:15.106464 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:15.106307    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce129f17f306f446acc5a55e3211ccd8ef72181dfa098d00fee3bee80a064160"
Jul 28 12:41:19.026320 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.026287    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"]
Jul 28 12:41:19.026773 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.026594    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b927db3f-ec6c-49f6-86be-07dba2864b35" containerName="pull"
Jul 28 12:41:19.026773 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.026605    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b927db3f-ec6c-49f6-86be-07dba2864b35" containerName="pull"
Jul 28 12:41:19.026773 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.026621    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b927db3f-ec6c-49f6-86be-07dba2864b35" containerName="extract"
Jul 28 12:41:19.026773 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.026627    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b927db3f-ec6c-49f6-86be-07dba2864b35" containerName="extract"
Jul 28 12:41:19.026773 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.026635    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b927db3f-ec6c-49f6-86be-07dba2864b35" containerName="util"
Jul 28 12:41:19.026773 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.026641    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b927db3f-ec6c-49f6-86be-07dba2864b35" containerName="util"
Jul 28 12:41:19.026773 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.026685    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b927db3f-ec6c-49f6-86be-07dba2864b35" containerName="extract"
Jul 28 12:41:19.078540 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.078507    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"]
Jul 28 12:41:19.078697 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.078618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"
Jul 28 12:41:19.081914 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.081884    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 12:41:19.081914 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.081899    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-ljmts\""
Jul 28 12:41:19.082090 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.081919    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 12:41:19.082281 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.082268    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 12:41:19.134961 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.134930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rbwl\" (UniqueName: \"kubernetes.io/projected/6681a6af-01a6-48ad-a3df-60940bb0d1a1-kube-api-access-2rbwl\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv\" (UID: \"6681a6af-01a6-48ad-a3df-60940bb0d1a1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"
Jul 28 12:41:19.135135 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.135003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6681a6af-01a6-48ad-a3df-60940bb0d1a1-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv\" (UID: \"6681a6af-01a6-48ad-a3df-60940bb0d1a1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"
Jul 28 12:41:19.236038 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.236005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbwl\" (UniqueName: \"kubernetes.io/projected/6681a6af-01a6-48ad-a3df-60940bb0d1a1-kube-api-access-2rbwl\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv\" (UID: \"6681a6af-01a6-48ad-a3df-60940bb0d1a1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"
Jul 28 12:41:19.236210 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.236065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6681a6af-01a6-48ad-a3df-60940bb0d1a1-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv\" (UID: \"6681a6af-01a6-48ad-a3df-60940bb0d1a1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"
Jul 28 12:41:19.238388 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.238361    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6681a6af-01a6-48ad-a3df-60940bb0d1a1-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv\" (UID: \"6681a6af-01a6-48ad-a3df-60940bb0d1a1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"
Jul 28 12:41:19.252695 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.252670    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbwl\" (UniqueName: \"kubernetes.io/projected/6681a6af-01a6-48ad-a3df-60940bb0d1a1-kube-api-access-2rbwl\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv\" (UID: \"6681a6af-01a6-48ad-a3df-60940bb0d1a1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"
Jul 28 12:41:19.388738 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.388646    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"
Jul 28 12:41:19.571611 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:19.571586    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"]
Jul 28 12:41:19.574155 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:41:19.574124    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6681a6af_01a6_48ad_a3df_60940bb0d1a1.slice/crio-b28b36faa91dae49049d494e8676802793a6930839bee376a77adc2e89f8baf6 WatchSource:0}: Error finding container b28b36faa91dae49049d494e8676802793a6930839bee376a77adc2e89f8baf6: Status 404 returned error can't find the container with id b28b36faa91dae49049d494e8676802793a6930839bee376a77adc2e89f8baf6
Jul 28 12:41:20.129283 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:20.129252    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv" event={"ID":"6681a6af-01a6-48ad-a3df-60940bb0d1a1","Type":"ContainerStarted","Data":"b28b36faa91dae49049d494e8676802793a6930839bee376a77adc2e89f8baf6"}
Jul 28 12:41:23.144737 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.144708    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv" event={"ID":"6681a6af-01a6-48ad-a3df-60940bb0d1a1","Type":"ContainerStarted","Data":"29f3f0848dac65737728f356394f74fa4778770713278819ed9fc7964a9364bd"}
Jul 28 12:41:23.800845 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.800807    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-wzbw4"]
Jul 28 12:41:23.819781 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.819754    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:23.822861 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.822834    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-wzbw4"]
Jul 28 12:41:23.823698 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.823677    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-5scgj\""
Jul 28 12:41:23.823902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.823874    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 12:41:23.824323 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.824304    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 12:41:23.876163 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.876138    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-certificates\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:23.876303 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.876188    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxn8x\" (UniqueName: \"kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-kube-api-access-wxn8x\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:23.876303 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.876219    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-cabundle0\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:23.977529 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.977489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxn8x\" (UniqueName: \"kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-kube-api-access-wxn8x\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:23.977695 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.977539    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-cabundle0\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:23.977695 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.977634    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-certificates\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:23.977805 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:23.977739    2569 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 12:41:23.977805 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:23.977757    2569 secret.go:281] references non-existent secret key: ca.crt
Jul 28 12:41:23.977805 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:23.977769    2569 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 12:41:23.977805 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:23.977784    2569 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-wzbw4: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 12:41:23.978015 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:23.977841    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-certificates podName:1656c6d0-e9d5-47a2-b1c1-96dc253d13b9 nodeName:}" failed. No retries permitted until 2026-07-28 12:41:24.477822404 +0000 UTC m=+452.693339039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-certificates") pod "keda-operator-77f648d694-wzbw4" (UID: "1656c6d0-e9d5-47a2-b1c1-96dc253d13b9") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 12:41:23.978385 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.978361    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-cabundle0\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:23.990958 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:23.990934    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxn8x\" (UniqueName: \"kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-kube-api-access-wxn8x\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:24.149398 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.149323    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"
Jul 28 12:41:24.182302 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.182244    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv" podStartSLOduration=1.749872597 podStartE2EDuration="5.182225796s" totalImagesPullingTime="3.432353199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:41:24.17536087 +0000 UTC m=+452.390877550" watchObservedRunningTime="2026-07-28 12:41:24.182225796 +0000 UTC m=+452.397742451"
Jul 28 12:41:24.485751 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.485719    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-certificates\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:24.485905 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:24.485886    2569 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 12:41:24.485905 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:24.485904    2569 secret.go:281] references non-existent secret key: ca.crt
Jul 28 12:41:24.485975 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:24.485911    2569 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 12:41:24.485975 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:24.485922    2569 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-wzbw4: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 12:41:24.486062 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:24.486003    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-certificates podName:1656c6d0-e9d5-47a2-b1c1-96dc253d13b9 nodeName:}" failed. No retries permitted until 2026-07-28 12:41:25.485973184 +0000 UTC m=+453.701489817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-certificates") pod "keda-operator-77f648d694-wzbw4" (UID: "1656c6d0-e9d5-47a2-b1c1-96dc253d13b9") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 12:41:24.664202 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.664168    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"]
Jul 28 12:41:24.680264 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.680241    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:24.686688 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.686655    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 12:41:24.697247 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.697223    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"]
Jul 28 12:41:24.789181 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.789099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a378c2ba-2641-45d4-a410-f9ce3e18b9a2-certificates\") pod \"keda-metrics-apiserver-55557df8c-9g24z\" (UID: \"a378c2ba-2641-45d4-a410-f9ce3e18b9a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:24.789181 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.789146    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a378c2ba-2641-45d4-a410-f9ce3e18b9a2-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-9g24z\" (UID: \"a378c2ba-2641-45d4-a410-f9ce3e18b9a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:24.789357 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.789204    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2kf\" (UniqueName: \"kubernetes.io/projected/a378c2ba-2641-45d4-a410-f9ce3e18b9a2-kube-api-access-kg2kf\") pod \"keda-metrics-apiserver-55557df8c-9g24z\" (UID: \"a378c2ba-2641-45d4-a410-f9ce3e18b9a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:24.889755 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.889717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a378c2ba-2641-45d4-a410-f9ce3e18b9a2-certificates\") pod \"keda-metrics-apiserver-55557df8c-9g24z\" (UID: \"a378c2ba-2641-45d4-a410-f9ce3e18b9a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:24.889946 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.889769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a378c2ba-2641-45d4-a410-f9ce3e18b9a2-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-9g24z\" (UID: \"a378c2ba-2641-45d4-a410-f9ce3e18b9a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:24.889946 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.889816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2kf\" (UniqueName: \"kubernetes.io/projected/a378c2ba-2641-45d4-a410-f9ce3e18b9a2-kube-api-access-kg2kf\") pod \"keda-metrics-apiserver-55557df8c-9g24z\" (UID: \"a378c2ba-2641-45d4-a410-f9ce3e18b9a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:24.890192 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.890169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a378c2ba-2641-45d4-a410-f9ce3e18b9a2-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-9g24z\" (UID: \"a378c2ba-2641-45d4-a410-f9ce3e18b9a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:24.892237 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.892213    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a378c2ba-2641-45d4-a410-f9ce3e18b9a2-certificates\") pod \"keda-metrics-apiserver-55557df8c-9g24z\" (UID: \"a378c2ba-2641-45d4-a410-f9ce3e18b9a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:24.908341 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.908311    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2kf\" (UniqueName: \"kubernetes.io/projected/a378c2ba-2641-45d4-a410-f9ce3e18b9a2-kube-api-access-kg2kf\") pod \"keda-metrics-apiserver-55557df8c-9g24z\" (UID: \"a378c2ba-2641-45d4-a410-f9ce3e18b9a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:24.990234 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:24.990205    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:25.040232 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.040154    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-wst9c"]
Jul 28 12:41:25.049490 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.049462    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:41:25.053895 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.053875    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 12:41:25.076870 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.076558    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-wst9c"]
Jul 28 12:41:25.092066 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.091298    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5810a3a5-345b-4181-b9db-ab428ffb2769-certificates\") pod \"keda-admission-56f9798bb4-wst9c\" (UID: \"5810a3a5-345b-4181-b9db-ab428ffb2769\") " pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:41:25.092066 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.091345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrprz\" (UniqueName: \"kubernetes.io/projected/5810a3a5-345b-4181-b9db-ab428ffb2769-kube-api-access-wrprz\") pod \"keda-admission-56f9798bb4-wst9c\" (UID: \"5810a3a5-345b-4181-b9db-ab428ffb2769\") " pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:41:25.179892 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:41:25.179861    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda378c2ba_2641_45d4_a410_f9ce3e18b9a2.slice/crio-fdf82943eafa7c5bca5e5a08c692a4e63bef722451484a989f9d90accee13ce4 WatchSource:0}: Error finding container fdf82943eafa7c5bca5e5a08c692a4e63bef722451484a989f9d90accee13ce4: Status 404 returned error can't find the container with id fdf82943eafa7c5bca5e5a08c692a4e63bef722451484a989f9d90accee13ce4
Jul 28 12:41:25.180265 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.180244    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"]
Jul 28 12:41:25.194693 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.193316    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5810a3a5-345b-4181-b9db-ab428ffb2769-certificates\") pod \"keda-admission-56f9798bb4-wst9c\" (UID: \"5810a3a5-345b-4181-b9db-ab428ffb2769\") " pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:41:25.194693 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.193374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrprz\" (UniqueName: \"kubernetes.io/projected/5810a3a5-345b-4181-b9db-ab428ffb2769-kube-api-access-wrprz\") pod \"keda-admission-56f9798bb4-wst9c\" (UID: \"5810a3a5-345b-4181-b9db-ab428ffb2769\") " pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:41:25.194693 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:25.193700    2569 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 28 12:41:25.194693 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:25.193721    2569 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-wst9c: secret "keda-admission-webhooks-certs" not found
Jul 28 12:41:25.194693 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:41:25.193769    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5810a3a5-345b-4181-b9db-ab428ffb2769-certificates podName:5810a3a5-345b-4181-b9db-ab428ffb2769 nodeName:}" failed. No retries permitted until 2026-07-28 12:41:25.693750927 +0000 UTC m=+453.909267563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5810a3a5-345b-4181-b9db-ab428ffb2769-certificates") pod "keda-admission-56f9798bb4-wst9c" (UID: "5810a3a5-345b-4181-b9db-ab428ffb2769") : secret "keda-admission-webhooks-certs" not found
Jul 28 12:41:25.203670 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.203636    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrprz\" (UniqueName: \"kubernetes.io/projected/5810a3a5-345b-4181-b9db-ab428ffb2769-kube-api-access-wrprz\") pod \"keda-admission-56f9798bb4-wst9c\" (UID: \"5810a3a5-345b-4181-b9db-ab428ffb2769\") " pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:41:25.496579 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.496547    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-certificates\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:25.499188 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.499156    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1656c6d0-e9d5-47a2-b1c1-96dc253d13b9-certificates\") pod \"keda-operator-77f648d694-wzbw4\" (UID: \"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9\") " pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:25.631046 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.630941    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:25.699128 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.699001    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5810a3a5-345b-4181-b9db-ab428ffb2769-certificates\") pod \"keda-admission-56f9798bb4-wst9c\" (UID: \"5810a3a5-345b-4181-b9db-ab428ffb2769\") " pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:41:25.702104 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.702078    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5810a3a5-345b-4181-b9db-ab428ffb2769-certificates\") pod \"keda-admission-56f9798bb4-wst9c\" (UID: \"5810a3a5-345b-4181-b9db-ab428ffb2769\") " pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:41:25.782543 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.782522    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-wzbw4"]
Jul 28 12:41:25.785099 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:41:25.785073    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1656c6d0_e9d5_47a2_b1c1_96dc253d13b9.slice/crio-5fe4a9ee024cc5d135a03a391c7581d18451f52d7d69c4963e41a0363d9b4586 WatchSource:0}: Error finding container 5fe4a9ee024cc5d135a03a391c7581d18451f52d7d69c4963e41a0363d9b4586: Status 404 returned error can't find the container with id 5fe4a9ee024cc5d135a03a391c7581d18451f52d7d69c4963e41a0363d9b4586
Jul 28 12:41:25.963544 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:25.963515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:41:26.122178 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:26.122147    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-wst9c"]
Jul 28 12:41:26.126490 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:41:26.126456    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5810a3a5_345b_4181_b9db_ab428ffb2769.slice/crio-5f99240fe87a50aab1715170a328f49bff3d1d7b7bdaff9b521102320dcb2841 WatchSource:0}: Error finding container 5f99240fe87a50aab1715170a328f49bff3d1d7b7bdaff9b521102320dcb2841: Status 404 returned error can't find the container with id 5f99240fe87a50aab1715170a328f49bff3d1d7b7bdaff9b521102320dcb2841
Jul 28 12:41:26.163469 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:26.163428    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-wzbw4" event={"ID":"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9","Type":"ContainerStarted","Data":"5fe4a9ee024cc5d135a03a391c7581d18451f52d7d69c4963e41a0363d9b4586"}
Jul 28 12:41:26.164634 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:26.164606    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-wst9c" event={"ID":"5810a3a5-345b-4181-b9db-ab428ffb2769","Type":"ContainerStarted","Data":"5f99240fe87a50aab1715170a328f49bff3d1d7b7bdaff9b521102320dcb2841"}
Jul 28 12:41:26.165928 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:26.165901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z" event={"ID":"a378c2ba-2641-45d4-a410-f9ce3e18b9a2","Type":"ContainerStarted","Data":"fdf82943eafa7c5bca5e5a08c692a4e63bef722451484a989f9d90accee13ce4"}
Jul 28 12:41:30.188551 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:30.188509    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z" event={"ID":"a378c2ba-2641-45d4-a410-f9ce3e18b9a2","Type":"ContainerStarted","Data":"3767ccf7cf96e5215c2f46fe115b71bb9deedea88967c1e732da212e71b1f9bd"}
Jul 28 12:41:30.189004 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:30.188583    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:30.189924 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:30.189891    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-wzbw4" event={"ID":"1656c6d0-e9d5-47a2-b1c1-96dc253d13b9","Type":"ContainerStarted","Data":"6dbc6629c89cb7fb3f960d4dca41a262d6ee620d7dacdca09751f88490a1901e"}
Jul 28 12:41:30.190049 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:30.189959    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:30.191097 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:30.191078    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-wst9c" event={"ID":"5810a3a5-345b-4181-b9db-ab428ffb2769","Type":"ContainerStarted","Data":"c8b8947a22e208c226ed60a0ea764886c930ef56d787113a420622b7f3495910"}
Jul 28 12:41:30.191212 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:30.191159    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:41:30.216148 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:30.216109    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z" podStartSLOduration=1.939831017 podStartE2EDuration="6.216094846s" totalImagesPullingTime="4.276263829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:41:30.214664491 +0000 UTC m=+458.430181170" watchObservedRunningTime="2026-07-28 12:41:30.216094846 +0000 UTC m=+458.431611499"
Jul 28 12:41:30.273698 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:30.273646    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-wzbw4" podStartSLOduration=3.589562666 podStartE2EDuration="7.273629778s" totalImagesPullingTime="3.684067112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:41:30.248575278 +0000 UTC m=+458.464091930" watchObservedRunningTime="2026-07-28 12:41:30.273629778 +0000 UTC m=+458.489146443"
Jul 28 12:41:30.274873 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:30.274609    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-wst9c" podStartSLOduration=1.943586998 podStartE2EDuration="5.274596895s" totalImagesPullingTime="3.331009897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:41:30.27353523 +0000 UTC m=+458.489051882" watchObservedRunningTime="2026-07-28 12:41:30.274596895 +0000 UTC m=+458.490113555"
Jul 28 12:41:31.545197 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.545160    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-847fd8d496-twwm9"]
Jul 28 12:41:31.547515 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.547497    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.561393 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.561368    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847fd8d496-twwm9"]
Jul 28 12:41:31.655686 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.655654    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-console-serving-cert\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.655686 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.655687    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-console-config\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.655904 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.655708    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-trusted-ca-bundle\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.655904 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.655759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-oauth-serving-cert\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.655904 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.655784    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-service-ca\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.655904 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.655799    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-console-oauth-config\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.655904 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.655851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tk95\" (UniqueName: \"kubernetes.io/projected/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-kube-api-access-7tk95\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.756415 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.756383    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-trusted-ca-bundle\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.756605 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.756431    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-oauth-serving-cert\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.756605 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.756526    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-service-ca\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.756605 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.756548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-console-oauth-config\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.756785 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.756676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tk95\" (UniqueName: \"kubernetes.io/projected/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-kube-api-access-7tk95\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.756785 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.756746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-console-serving-cert\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.756881 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.756783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-console-config\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.757338 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.757315    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-trusted-ca-bundle\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.757338 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.757330    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-service-ca\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.757484 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.757408    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-oauth-serving-cert\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.757565 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.757544    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-console-config\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.759140 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.759117    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-console-serving-cert\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.759224 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.759150    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-console-oauth-config\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.766794 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.766774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tk95\" (UniqueName: \"kubernetes.io/projected/0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1-kube-api-access-7tk95\") pod \"console-847fd8d496-twwm9\" (UID: \"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1\") " pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.856374 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.856281    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:31.986092 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:31.986066    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847fd8d496-twwm9"]
Jul 28 12:41:31.988140 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:41:31.988115    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4a497c_bda8_4a7c_8418_ac7dd8d27ed1.slice/crio-102614ceaf3eaa520cb30c57dd9496e47f296dc1a1198841f2d8a43fefd40875 WatchSource:0}: Error finding container 102614ceaf3eaa520cb30c57dd9496e47f296dc1a1198841f2d8a43fefd40875: Status 404 returned error can't find the container with id 102614ceaf3eaa520cb30c57dd9496e47f296dc1a1198841f2d8a43fefd40875
Jul 28 12:41:32.202008 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:32.201899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847fd8d496-twwm9" event={"ID":"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1","Type":"ContainerStarted","Data":"9cbdb61cfc6647f702e7cd8535f158550dd63409be01506454505f37d9278c05"}
Jul 28 12:41:32.202008 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:32.201955    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847fd8d496-twwm9" event={"ID":"0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1","Type":"ContainerStarted","Data":"102614ceaf3eaa520cb30c57dd9496e47f296dc1a1198841f2d8a43fefd40875"}
Jul 28 12:41:32.221105 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:32.221052    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-847fd8d496-twwm9" podStartSLOduration=1.221033147 podStartE2EDuration="1.221033147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:41:32.219660635 +0000 UTC m=+460.435177288" watchObservedRunningTime="2026-07-28 12:41:32.221033147 +0000 UTC m=+460.436549801"
Jul 28 12:41:41.199945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:41.199915    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9g24z"
Jul 28 12:41:41.856763 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:41.856711    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:41.856951 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:41.856785    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:41.861527 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:41.861506    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:42.252690 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:42.252657    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-847fd8d496-twwm9"
Jul 28 12:41:42.299591 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:42.299559    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8499c78db4-bbr8h"]
Jul 28 12:41:45.157416 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:45.157383    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gh9xv"
Jul 28 12:41:51.198713 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:51.198681    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-wzbw4"
Jul 28 12:41:51.199322 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:41:51.199012    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-wst9c"
Jul 28 12:42:07.319707 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.319649    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8499c78db4-bbr8h" podUID="c78a9ce1-a9d9-4ec5-9500-e302915e34eb" containerName="console" containerID="cri-o://6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486" gracePeriod=15
Jul 28 12:42:07.356648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.355903    2569 patch_prober.go:28] interesting pod/console-8499c78db4-bbr8h container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused" start-of-body=
Jul 28 12:42:07.356933 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.356873    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-8499c78db4-bbr8h" podUID="c78a9ce1-a9d9-4ec5-9500-e302915e34eb" containerName="console" probeResult="failure" output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused"
Jul 28 12:42:07.568468 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.568445    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:42:07.663982 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.663893    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7wdv\" (UniqueName: \"kubernetes.io/projected/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-kube-api-access-p7wdv\") pod \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") "
Jul 28 12:42:07.663982 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.663931    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-serving-cert\") pod \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") "
Jul 28 12:42:07.663982 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.663949    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-config\") pod \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") "
Jul 28 12:42:07.663982 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.663975    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-service-ca\") pod \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") "
Jul 28 12:42:07.664303 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.664065    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-oauth-config\") pod \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") "
Jul 28 12:42:07.664303 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.664099    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-oauth-serving-cert\") pod \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") "
Jul 28 12:42:07.664303 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.664139    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-trusted-ca-bundle\") pod \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\" (UID: \"c78a9ce1-a9d9-4ec5-9500-e302915e34eb\") "
Jul 28 12:42:07.664497 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.664471    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-config" (OuterVolumeSpecName: "console-config") pod "c78a9ce1-a9d9-4ec5-9500-e302915e34eb" (UID: "c78a9ce1-a9d9-4ec5-9500-e302915e34eb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:42:07.664705 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.664666    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c78a9ce1-a9d9-4ec5-9500-e302915e34eb" (UID: "c78a9ce1-a9d9-4ec5-9500-e302915e34eb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:42:07.664803 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.664776    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c78a9ce1-a9d9-4ec5-9500-e302915e34eb" (UID: "c78a9ce1-a9d9-4ec5-9500-e302915e34eb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:42:07.664864 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.664781    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-service-ca" (OuterVolumeSpecName: "service-ca") pod "c78a9ce1-a9d9-4ec5-9500-e302915e34eb" (UID: "c78a9ce1-a9d9-4ec5-9500-e302915e34eb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:42:07.666264 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.666230    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-kube-api-access-p7wdv" (OuterVolumeSpecName: "kube-api-access-p7wdv") pod "c78a9ce1-a9d9-4ec5-9500-e302915e34eb" (UID: "c78a9ce1-a9d9-4ec5-9500-e302915e34eb"). InnerVolumeSpecName "kube-api-access-p7wdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:42:07.666264 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.666253    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c78a9ce1-a9d9-4ec5-9500-e302915e34eb" (UID: "c78a9ce1-a9d9-4ec5-9500-e302915e34eb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:42:07.666412 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.666233    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c78a9ce1-a9d9-4ec5-9500-e302915e34eb" (UID: "c78a9ce1-a9d9-4ec5-9500-e302915e34eb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:42:07.765177 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.765136    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-oauth-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:42:07.765177 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.765167    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-trusted-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:42:07.765177 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.765178    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7wdv\" (UniqueName: \"kubernetes.io/projected/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-kube-api-access-p7wdv\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:42:07.765177 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.765188    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:42:07.765440 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.765197    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:42:07.765440 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.765205    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:42:07.765440 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:07.765213    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c78a9ce1-a9d9-4ec5-9500-e302915e34eb-console-oauth-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:42:08.362001 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:08.361953    2569 generic.go:356] "Generic (PLEG): container finished" podID="c78a9ce1-a9d9-4ec5-9500-e302915e34eb" containerID="6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486" exitCode=0
Jul 28 12:42:08.362430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:08.362015    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8499c78db4-bbr8h"
Jul 28 12:42:08.362430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:08.362037    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8499c78db4-bbr8h" event={"ID":"c78a9ce1-a9d9-4ec5-9500-e302915e34eb","Type":"ContainerDied","Data":"6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486"}
Jul 28 12:42:08.362430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:08.362088    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8499c78db4-bbr8h" event={"ID":"c78a9ce1-a9d9-4ec5-9500-e302915e34eb","Type":"ContainerDied","Data":"1f9beb8d8e53f75b808b838897dafae344591c494fd4af2aa451038617363285"}
Jul 28 12:42:08.362430 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:08.362108    2569 scope.go:117] "RemoveContainer" containerID="6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486"
Jul 28 12:42:08.373452 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:08.373429    2569 scope.go:117] "RemoveContainer" containerID="6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486"
Jul 28 12:42:08.373744 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:42:08.373724    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486\": container with ID starting with 6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486 not found: ID does not exist" containerID="6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486"
Jul 28 12:42:08.373783 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:08.373755    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486"} err="failed to get container status \"6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486\": rpc error: code = NotFound desc = could not find container \"6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486\": container with ID starting with 6fca7ecda74ac1886d1f26c337c6d5689f189c88b9484b1b0fbed4f6125cc486 not found: ID does not exist"
Jul 28 12:42:08.378620 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:08.378595    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8499c78db4-bbr8h"]
Jul 28 12:42:08.384123 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:08.384102    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8499c78db4-bbr8h"]
Jul 28 12:42:10.251279 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:10.251247    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c78a9ce1-a9d9-4ec5-9500-e302915e34eb" path="/var/lib/kubelet/pods/c78a9ce1-a9d9-4ec5-9500-e302915e34eb/volumes"
Jul 28 12:42:43.328402 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.328362    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-f9d98"]
Jul 28 12:42:43.328841 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.328710    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78a9ce1-a9d9-4ec5-9500-e302915e34eb" containerName="console"
Jul 28 12:42:43.328841 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.328721    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78a9ce1-a9d9-4ec5-9500-e302915e34eb" containerName="console"
Jul 28 12:42:43.328841 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.328775    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c78a9ce1-a9d9-4ec5-9500-e302915e34eb" containerName="console"
Jul 28 12:42:43.330806 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.330787    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:42:43.333548 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.333526    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 12:42:43.334375 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.334353    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 12:42:43.334481 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.334353    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 12:42:43.334481 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.334383    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-ns4kq\""
Jul 28 12:42:43.353419 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.353390    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-f9d98"]
Jul 28 12:42:43.468360 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.468320    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wbdp\" (UniqueName: \"kubernetes.io/projected/9491f529-30d3-442c-90be-082c56aa8994-kube-api-access-6wbdp\") pod \"llmisvc-controller-manager-654c557856-f9d98\" (UID: \"9491f529-30d3-442c-90be-082c56aa8994\") " pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:42:43.468360 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.468368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9491f529-30d3-442c-90be-082c56aa8994-cert\") pod \"llmisvc-controller-manager-654c557856-f9d98\" (UID: \"9491f529-30d3-442c-90be-082c56aa8994\") " pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:42:43.569542 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.569510    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbdp\" (UniqueName: \"kubernetes.io/projected/9491f529-30d3-442c-90be-082c56aa8994-kube-api-access-6wbdp\") pod \"llmisvc-controller-manager-654c557856-f9d98\" (UID: \"9491f529-30d3-442c-90be-082c56aa8994\") " pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:42:43.569703 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.569549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9491f529-30d3-442c-90be-082c56aa8994-cert\") pod \"llmisvc-controller-manager-654c557856-f9d98\" (UID: \"9491f529-30d3-442c-90be-082c56aa8994\") " pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:42:43.569703 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:42:43.569681    2569 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 28 12:42:43.569773 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:42:43.569742    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9491f529-30d3-442c-90be-082c56aa8994-cert podName:9491f529-30d3-442c-90be-082c56aa8994 nodeName:}" failed. No retries permitted until 2026-07-28 12:42:44.069724565 +0000 UTC m=+532.285241196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9491f529-30d3-442c-90be-082c56aa8994-cert") pod "llmisvc-controller-manager-654c557856-f9d98" (UID: "9491f529-30d3-442c-90be-082c56aa8994") : secret "llmisvc-webhook-server-cert" not found
Jul 28 12:42:43.577979 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:43.577947    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbdp\" (UniqueName: \"kubernetes.io/projected/9491f529-30d3-442c-90be-082c56aa8994-kube-api-access-6wbdp\") pod \"llmisvc-controller-manager-654c557856-f9d98\" (UID: \"9491f529-30d3-442c-90be-082c56aa8994\") " pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:42:44.072298 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:44.072261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9491f529-30d3-442c-90be-082c56aa8994-cert\") pod \"llmisvc-controller-manager-654c557856-f9d98\" (UID: \"9491f529-30d3-442c-90be-082c56aa8994\") " pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:42:44.074593 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:44.074570    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9491f529-30d3-442c-90be-082c56aa8994-cert\") pod \"llmisvc-controller-manager-654c557856-f9d98\" (UID: \"9491f529-30d3-442c-90be-082c56aa8994\") " pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:42:44.240177 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:44.240102    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:42:44.381520 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:44.381406    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-f9d98"]
Jul 28 12:42:44.384193 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:42:44.384162    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9491f529_30d3_442c_90be_082c56aa8994.slice/crio-0e138d060a877103eba2502ae26cc7acd00f787ed2a56998dce9f18b076b6f85 WatchSource:0}: Error finding container 0e138d060a877103eba2502ae26cc7acd00f787ed2a56998dce9f18b076b6f85: Status 404 returned error can't find the container with id 0e138d060a877103eba2502ae26cc7acd00f787ed2a56998dce9f18b076b6f85
Jul 28 12:42:44.528448 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:44.528412    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-f9d98" event={"ID":"9491f529-30d3-442c-90be-082c56aa8994","Type":"ContainerStarted","Data":"0e138d060a877103eba2502ae26cc7acd00f787ed2a56998dce9f18b076b6f85"}
Jul 28 12:42:47.544527 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:47.544489    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-f9d98" event={"ID":"9491f529-30d3-442c-90be-082c56aa8994","Type":"ContainerStarted","Data":"01a0ef88eb76870233d0a24ccbd2de25be85a8dfa6cc838b1908fffd5d7d001a"}
Jul 28 12:42:47.544927 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:47.544627    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:42:47.566912 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:42:47.566839    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-f9d98" podStartSLOduration=2.504473417 podStartE2EDuration="4.566821403s" totalImagesPullingTime="2.062347986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:42:47.56427374 +0000 UTC m=+535.779790394" watchObservedRunningTime="2026-07-28 12:42:47.566821403 +0000 UTC m=+535.782338054"
Jul 28 12:43:18.551966 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:43:18.551933    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-f9d98"
Jul 28 12:44:02.328486 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.328457    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-qwhkq"]
Jul 28 12:44:02.331727 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.331711    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-qwhkq"
Jul 28 12:44:02.334440 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.334417    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 12:44:02.334693 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.334674    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-fc8d2\""
Jul 28 12:44:02.341501 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.341477    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-qwhkq"]
Jul 28 12:44:02.345902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.345881    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-dchcx"]
Jul 28 12:44:02.349165 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.349148    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-dchcx"
Jul 28 12:44:02.351886 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.351864    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 12:44:02.352009 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.351872    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-4dwr9\""
Jul 28 12:44:02.360558 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.360538    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-dchcx"]
Jul 28 12:44:02.427553 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.427527    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdzmt\" (UniqueName: \"kubernetes.io/projected/1b26e8e6-91ce-4cb5-b58c-a60fb3e55228-kube-api-access-mdzmt\") pod \"odh-model-controller-75d9cdbddf-dchcx\" (UID: \"1b26e8e6-91ce-4cb5-b58c-a60fb3e55228\") " pod="kserve/odh-model-controller-75d9cdbddf-dchcx"
Jul 28 12:44:02.427711 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.427563    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b26e8e6-91ce-4cb5-b58c-a60fb3e55228-cert\") pod \"odh-model-controller-75d9cdbddf-dchcx\" (UID: \"1b26e8e6-91ce-4cb5-b58c-a60fb3e55228\") " pod="kserve/odh-model-controller-75d9cdbddf-dchcx"
Jul 28 12:44:02.427711 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.427582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtsvw\" (UniqueName: \"kubernetes.io/projected/1b1e00c1-36cf-4a6d-a96d-26e28b14b418-kube-api-access-vtsvw\") pod \"model-serving-api-7b954ffc95-qwhkq\" (UID: \"1b1e00c1-36cf-4a6d-a96d-26e28b14b418\") " pod="kserve/model-serving-api-7b954ffc95-qwhkq"
Jul 28 12:44:02.427711 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.427598    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1e00c1-36cf-4a6d-a96d-26e28b14b418-tls-certs\") pod \"model-serving-api-7b954ffc95-qwhkq\" (UID: \"1b1e00c1-36cf-4a6d-a96d-26e28b14b418\") " pod="kserve/model-serving-api-7b954ffc95-qwhkq"
Jul 28 12:44:02.528284 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.528245    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzmt\" (UniqueName: \"kubernetes.io/projected/1b26e8e6-91ce-4cb5-b58c-a60fb3e55228-kube-api-access-mdzmt\") pod \"odh-model-controller-75d9cdbddf-dchcx\" (UID: \"1b26e8e6-91ce-4cb5-b58c-a60fb3e55228\") " pod="kserve/odh-model-controller-75d9cdbddf-dchcx"
Jul 28 12:44:02.528458 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.528291    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b26e8e6-91ce-4cb5-b58c-a60fb3e55228-cert\") pod \"odh-model-controller-75d9cdbddf-dchcx\" (UID: \"1b26e8e6-91ce-4cb5-b58c-a60fb3e55228\") " pod="kserve/odh-model-controller-75d9cdbddf-dchcx"
Jul 28 12:44:02.528458 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.528320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtsvw\" (UniqueName: \"kubernetes.io/projected/1b1e00c1-36cf-4a6d-a96d-26e28b14b418-kube-api-access-vtsvw\") pod \"model-serving-api-7b954ffc95-qwhkq\" (UID: \"1b1e00c1-36cf-4a6d-a96d-26e28b14b418\") " pod="kserve/model-serving-api-7b954ffc95-qwhkq"
Jul 28 12:44:02.528458 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.528348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1e00c1-36cf-4a6d-a96d-26e28b14b418-tls-certs\") pod \"model-serving-api-7b954ffc95-qwhkq\" (UID: \"1b1e00c1-36cf-4a6d-a96d-26e28b14b418\") " pod="kserve/model-serving-api-7b954ffc95-qwhkq"
Jul 28 12:44:02.530805 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.530781    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b26e8e6-91ce-4cb5-b58c-a60fb3e55228-cert\") pod \"odh-model-controller-75d9cdbddf-dchcx\" (UID: \"1b26e8e6-91ce-4cb5-b58c-a60fb3e55228\") " pod="kserve/odh-model-controller-75d9cdbddf-dchcx"
Jul 28 12:44:02.530911 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.530848    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1e00c1-36cf-4a6d-a96d-26e28b14b418-tls-certs\") pod \"model-serving-api-7b954ffc95-qwhkq\" (UID: \"1b1e00c1-36cf-4a6d-a96d-26e28b14b418\") " pod="kserve/model-serving-api-7b954ffc95-qwhkq"
Jul 28 12:44:02.537786 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.537761    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzmt\" (UniqueName: \"kubernetes.io/projected/1b26e8e6-91ce-4cb5-b58c-a60fb3e55228-kube-api-access-mdzmt\") pod \"odh-model-controller-75d9cdbddf-dchcx\" (UID: \"1b26e8e6-91ce-4cb5-b58c-a60fb3e55228\") " pod="kserve/odh-model-controller-75d9cdbddf-dchcx"
Jul 28 12:44:02.537947 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.537928    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtsvw\" (UniqueName: \"kubernetes.io/projected/1b1e00c1-36cf-4a6d-a96d-26e28b14b418-kube-api-access-vtsvw\") pod \"model-serving-api-7b954ffc95-qwhkq\" (UID: \"1b1e00c1-36cf-4a6d-a96d-26e28b14b418\") " pod="kserve/model-serving-api-7b954ffc95-qwhkq"
Jul 28 12:44:02.642255 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.642170    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-qwhkq"
Jul 28 12:44:02.659365 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.659338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-dchcx"
Jul 28 12:44:02.776656 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.776621    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-qwhkq"]
Jul 28 12:44:02.779098 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:44:02.779072    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1e00c1_36cf_4a6d_a96d_26e28b14b418.slice/crio-4e04177c5173440956909fa1726f6983b1a426c711c366a0ef7f3f3b7663ca83 WatchSource:0}: Error finding container 4e04177c5173440956909fa1726f6983b1a426c711c366a0ef7f3f3b7663ca83: Status 404 returned error can't find the container with id 4e04177c5173440956909fa1726f6983b1a426c711c366a0ef7f3f3b7663ca83
Jul 28 12:44:02.798204 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.798182    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-dchcx"]
Jul 28 12:44:02.800295 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:44:02.800268    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b26e8e6_91ce_4cb5_b58c_a60fb3e55228.slice/crio-a4c81cbd8642341d40f3a80fa9acfba7b4c4c541e3fa6d6c9539a5ae71e3ce3a WatchSource:0}: Error finding container a4c81cbd8642341d40f3a80fa9acfba7b4c4c541e3fa6d6c9539a5ae71e3ce3a: Status 404 returned error can't find the container with id a4c81cbd8642341d40f3a80fa9acfba7b4c4c541e3fa6d6c9539a5ae71e3ce3a
Jul 28 12:44:02.882748 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.882712    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-dchcx" event={"ID":"1b26e8e6-91ce-4cb5-b58c-a60fb3e55228","Type":"ContainerStarted","Data":"a4c81cbd8642341d40f3a80fa9acfba7b4c4c541e3fa6d6c9539a5ae71e3ce3a"}
Jul 28 12:44:02.883793 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:02.883767    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-qwhkq" event={"ID":"1b1e00c1-36cf-4a6d-a96d-26e28b14b418","Type":"ContainerStarted","Data":"4e04177c5173440956909fa1726f6983b1a426c711c366a0ef7f3f3b7663ca83"}
Jul 28 12:44:06.910217 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:06.910172    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-dchcx" event={"ID":"1b26e8e6-91ce-4cb5-b58c-a60fb3e55228","Type":"ContainerStarted","Data":"1fc7fd2eae3ebf2c9ceb50b66863798e932f434ba0b1ca1363a6598f2864b0c5"}
Jul 28 12:44:06.910664 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:06.910274    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-dchcx"
Jul 28 12:44:06.911606 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:06.911584    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-qwhkq" event={"ID":"1b1e00c1-36cf-4a6d-a96d-26e28b14b418","Type":"ContainerStarted","Data":"d22a092ac269ff0118f40212dd00a3ccbc1871f6e58168748dd087f286f4b0aa"}
Jul 28 12:44:06.911726 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:06.911712    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-qwhkq"
Jul 28 12:44:06.926440 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:06.926385    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-dchcx" podStartSLOduration=1.2633535 podStartE2EDuration="4.926367505s" totalImagesPullingTime="3.663014005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:44:06.925182179 +0000 UTC m=+615.140698842" watchObservedRunningTime="2026-07-28 12:44:06.926367505 +0000 UTC m=+615.141884157"
Jul 28 12:44:06.940679 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:06.940634    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-qwhkq" podStartSLOduration=1.30895696 podStartE2EDuration="4.940617077s" totalImagesPullingTime="3.631660117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:44:06.940128543 +0000 UTC m=+615.155645199" watchObservedRunningTime="2026-07-28 12:44:06.940617077 +0000 UTC m=+615.156133728"
Jul 28 12:44:17.919503 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:17.919434    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-dchcx"
Jul 28 12:44:17.921315 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:44:17.921293    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-qwhkq"
Jul 28 12:47:47.130460 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.130425    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"]
Jul 28 12:47:47.134030 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.134011    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.138299 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.138271    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e5cf3-predictor-serving-cert\""
Jul 28 12:47:47.138408 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.138323    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 12:47:47.138471 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.138457    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 12:47:47.138528 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.138477    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rtw6d\""
Jul 28 12:47:47.138598 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.138584    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e5cf3-kube-rbac-proxy-sar-config\""
Jul 28 12:47:47.138702 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.138686    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 12:47:47.138770 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.138745    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 12:47:47.167779 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.167753    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"]
Jul 28 12:47:47.271206 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.271176    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.271368 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.271217    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-proxy-tls\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.271368 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.271236    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.271368 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.271263    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664h9\" (UniqueName: \"kubernetes.io/projected/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-api-access-664h9\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.271368 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.271289    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.372131 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.372097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.372284 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.372140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-proxy-tls\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.372284 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.372160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.372284 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.372179    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664h9\" (UniqueName: \"kubernetes.io/projected/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-api-access-664h9\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.372284 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.372199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.372284 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:47:47.372270    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-serving-cert: secret "isvc-xgboost-graph-raw-e5cf3-predictor-serving-cert" not found
Jul 28 12:47:47.372458 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:47:47.372370    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-proxy-tls podName:d34136b6-a566-4ef9-9d64-ecb48eda1fd6 nodeName:}" failed. No retries permitted until 2026-07-28 12:47:47.872350165 +0000 UTC m=+836.087866804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-proxy-tls") pod "isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" (UID: "d34136b6-a566-4ef9-9d64-ecb48eda1fd6") : secret "isvc-xgboost-graph-raw-e5cf3-predictor-serving-cert" not found
Jul 28 12:47:47.372545 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.372520    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.372857 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.372840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.372919 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.372901    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.385626 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.385572    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-664h9\" (UniqueName: \"kubernetes.io/projected/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-api-access-664h9\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.876624 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.876587    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-proxy-tls\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:47.878902 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:47.878883    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-proxy-tls\") pod \"isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:48.044543 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:48.044501    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:47:48.188144 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:48.188119    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"]
Jul 28 12:47:48.190526 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:47:48.190499    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34136b6_a566_4ef9_9d64_ecb48eda1fd6.slice/crio-90700a28053606e656d92fdc953c71adf7c8a16634e831d5139019f90a0841a4 WatchSource:0}: Error finding container 90700a28053606e656d92fdc953c71adf7c8a16634e831d5139019f90a0841a4: Status 404 returned error can't find the container with id 90700a28053606e656d92fdc953c71adf7c8a16634e831d5139019f90a0841a4
Jul 28 12:47:48.192502 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:48.192481    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:47:48.902493 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:48.902447    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" event={"ID":"d34136b6-a566-4ef9-9d64-ecb48eda1fd6","Type":"ContainerStarted","Data":"90700a28053606e656d92fdc953c71adf7c8a16634e831d5139019f90a0841a4"}
Jul 28 12:47:51.918498 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:51.918456    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" event={"ID":"d34136b6-a566-4ef9-9d64-ecb48eda1fd6","Type":"ContainerStarted","Data":"32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886"}
Jul 28 12:47:55.938744 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:55.938667    2569 generic.go:356] "Generic (PLEG): container finished" podID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerID="32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886" exitCode=0
Jul 28 12:47:55.938744 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:47:55.938729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" event={"ID":"d34136b6-a566-4ef9-9d64-ecb48eda1fd6","Type":"ContainerDied","Data":"32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886"}
Jul 28 12:48:13.033378 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:13.033346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" event={"ID":"d34136b6-a566-4ef9-9d64-ecb48eda1fd6","Type":"ContainerStarted","Data":"184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8"}
Jul 28 12:48:15.045159 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:15.045130    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" event={"ID":"d34136b6-a566-4ef9-9d64-ecb48eda1fd6","Type":"ContainerStarted","Data":"ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3"}
Jul 28 12:48:15.045491 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:15.045381    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:48:15.045491 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:15.045447    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:48:15.047005 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:15.046959    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:48:15.065025 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:15.064965    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" podStartSLOduration=1.7543687719999999 podStartE2EDuration="28.064950946s" totalImagesPullingTime="22.522493052s" totalInitContainerRuntime="3.788089122s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:47:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:48:15.064247991 +0000 UTC m=+863.279764656" watchObservedRunningTime="2026-07-28 12:48:15.064950946 +0000 UTC m=+863.280467598"
Jul 28 12:48:16.049587 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:16.049543    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:48:21.054057 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:21.054025    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:48:21.054518 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:21.054492    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:48:31.054463 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:31.054413    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:48:41.054859 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:41.054813    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:48:51.055053 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:48:51.054937    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:49:01.055048 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:01.054984    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:49:11.055870 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:11.055840    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:49:37.351002 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.350953    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"]
Jul 28 12:49:37.351584 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.351262    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container" containerID="cri-o://184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8" gracePeriod=30
Jul 28 12:49:37.351584 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.351278    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kube-rbac-proxy" containerID="cri-o://ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3" gracePeriod=30
Jul 28 12:49:37.441242 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.441207    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"]
Jul 28 12:49:37.444866 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.444847    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.448578 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.448558    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-7f8c4-kube-rbac-proxy-sar-config\""
Jul 28 12:49:37.448704 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.448592    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-serving-cert\""
Jul 28 12:49:37.457667 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.457639    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"]
Jul 28 12:49:37.527599 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.527567    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.527599 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.527605    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.527867 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.527630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.527867 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.527717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpq74\" (UniqueName: \"kubernetes.io/projected/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-api-access-jpq74\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.527867 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.527765    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.628818 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.628731    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.628818 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.628767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.628818 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.628786    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.629155 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.628824    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpq74\" (UniqueName: \"kubernetes.io/projected/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-api-access-jpq74\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.629155 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.628839    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.629155 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:49:37.628895    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-7f8c4-predictor-serving-cert" not found
Jul 28 12:49:37.629155 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:49:37.628977    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-proxy-tls podName:12ec7f96-fb1d-45a1-b15b-36d1d129dc71 nodeName:}" failed. No retries permitted until 2026-07-28 12:49:38.128953509 +0000 UTC m=+946.344470162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" (UID: "12ec7f96-fb1d-45a1-b15b-36d1d129dc71") : secret "isvc-xgboost-graph-raw-hpa-7f8c4-predictor-serving-cert" not found
Jul 28 12:49:37.629393 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.629375    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.629467 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.629448    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.629621 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.629603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:37.638200 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:37.638179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpq74\" (UniqueName: \"kubernetes.io/projected/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-api-access-jpq74\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:38.134308 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:38.134269    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:38.136674 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:38.136648    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:38.354540 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:38.354499    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:38.419810 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:38.419776    2569 generic.go:356] "Generic (PLEG): container finished" podID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerID="ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3" exitCode=2
Jul 28 12:49:38.419968 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:38.419873    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" event={"ID":"d34136b6-a566-4ef9-9d64-ecb48eda1fd6","Type":"ContainerDied","Data":"ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3"}
Jul 28 12:49:38.493400 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:38.493372    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"]
Jul 28 12:49:38.495117 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:49:38.495094    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ec7f96_fb1d_45a1_b15b_36d1d129dc71.slice/crio-7cde39cf3ad7e864c0c5f6e6c6ef19d67a9500542b63a2edff4a236ac4eb0c9c WatchSource:0}: Error finding container 7cde39cf3ad7e864c0c5f6e6c6ef19d67a9500542b63a2edff4a236ac4eb0c9c: Status 404 returned error can't find the container with id 7cde39cf3ad7e864c0c5f6e6c6ef19d67a9500542b63a2edff4a236ac4eb0c9c
Jul 28 12:49:39.425702 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:39.425668    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" event={"ID":"12ec7f96-fb1d-45a1-b15b-36d1d129dc71","Type":"ContainerStarted","Data":"22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0"}
Jul 28 12:49:39.426079 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:39.425731    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" event={"ID":"12ec7f96-fb1d-45a1-b15b-36d1d129dc71","Type":"ContainerStarted","Data":"7cde39cf3ad7e864c0c5f6e6c6ef19d67a9500542b63a2edff4a236ac4eb0c9c"}
Jul 28 12:49:41.009323 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.009296    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:49:41.062885 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.062799    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-rbac-proxy-sar-config\") pod \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") "
Jul 28 12:49:41.062885 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.062865    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-664h9\" (UniqueName: \"kubernetes.io/projected/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-api-access-664h9\") pod \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") "
Jul 28 12:49:41.063144 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.062888    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-cabundle-cert\") pod \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") "
Jul 28 12:49:41.063144 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.062921    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-proxy-tls\") pod \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") "
Jul 28 12:49:41.063144 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.063038    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kserve-provision-location\") pod \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\" (UID: \"d34136b6-a566-4ef9-9d64-ecb48eda1fd6\") "
Jul 28 12:49:41.063622 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.063344    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d34136b6-a566-4ef9-9d64-ecb48eda1fd6" (UID: "d34136b6-a566-4ef9-9d64-ecb48eda1fd6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:49:41.063622 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.063371    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d34136b6-a566-4ef9-9d64-ecb48eda1fd6" (UID: "d34136b6-a566-4ef9-9d64-ecb48eda1fd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:49:41.063622 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.063386    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d34136b6-a566-4ef9-9d64-ecb48eda1fd6" (UID: "d34136b6-a566-4ef9-9d64-ecb48eda1fd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:49:41.065136 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.065112    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d34136b6-a566-4ef9-9d64-ecb48eda1fd6" (UID: "d34136b6-a566-4ef9-9d64-ecb48eda1fd6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:49:41.065245 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.065153    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-api-access-664h9" (OuterVolumeSpecName: "kube-api-access-664h9") pod "d34136b6-a566-4ef9-9d64-ecb48eda1fd6" (UID: "d34136b6-a566-4ef9-9d64-ecb48eda1fd6"). InnerVolumeSpecName "kube-api-access-664h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:49:41.163722 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.163684    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kserve-provision-location\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:49:41.163722 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.163716    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:49:41.163722 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.163730    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-664h9\" (UniqueName: \"kubernetes.io/projected/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-kube-api-access-664h9\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:49:41.163949 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.163740    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-cabundle-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:49:41.163949 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.163750    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d34136b6-a566-4ef9-9d64-ecb48eda1fd6-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:49:41.440110 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.440023    2569 generic.go:356] "Generic (PLEG): container finished" podID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerID="184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8" exitCode=0
Jul 28 12:49:41.440110 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.440095    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"
Jul 28 12:49:41.440284 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.440105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" event={"ID":"d34136b6-a566-4ef9-9d64-ecb48eda1fd6","Type":"ContainerDied","Data":"184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8"}
Jul 28 12:49:41.440284 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.440167    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5" event={"ID":"d34136b6-a566-4ef9-9d64-ecb48eda1fd6","Type":"ContainerDied","Data":"90700a28053606e656d92fdc953c71adf7c8a16634e831d5139019f90a0841a4"}
Jul 28 12:49:41.440284 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.440197    2569 scope.go:117] "RemoveContainer" containerID="ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3"
Jul 28 12:49:41.449327 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.449308    2569 scope.go:117] "RemoveContainer" containerID="184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8"
Jul 28 12:49:41.456964 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.456943    2569 scope.go:117] "RemoveContainer" containerID="32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886"
Jul 28 12:49:41.463648 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.463619    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"]
Jul 28 12:49:41.465243 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.465220    2569 scope.go:117] "RemoveContainer" containerID="ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3"
Jul 28 12:49:41.465621 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:49:41.465592    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3\": container with ID starting with ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3 not found: ID does not exist" containerID="ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3"
Jul 28 12:49:41.465856 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.465630    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3"} err="failed to get container status \"ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3\": rpc error: code = NotFound desc = could not find container \"ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3\": container with ID starting with ee1b05bd488c6af45235e8187e14d69df3d573d37bcc0aabe623240506e986a3 not found: ID does not exist"
Jul 28 12:49:41.465856 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.465662    2569 scope.go:117] "RemoveContainer" containerID="184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8"
Jul 28 12:49:41.466184 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:49:41.465953    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8\": container with ID starting with 184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8 not found: ID does not exist" containerID="184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8"
Jul 28 12:49:41.466184 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.466011    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8"} err="failed to get container status \"184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8\": rpc error: code = NotFound desc = could not find container \"184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8\": container with ID starting with 184e5284410074c67e1d5928270875cb32046aa90d6939233800e9b8c040e7f8 not found: ID does not exist"
Jul 28 12:49:41.466184 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.466030    2569 scope.go:117] "RemoveContainer" containerID="32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886"
Jul 28 12:49:41.466323 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:49:41.466298    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886\": container with ID starting with 32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886 not found: ID does not exist" containerID="32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886"
Jul 28 12:49:41.466372 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.466317    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886"} err="failed to get container status \"32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886\": rpc error: code = NotFound desc = could not find container \"32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886\": container with ID starting with 32863e8d5213622f101ddae636f87339ff082eaec3f436836b8d6f17722b9886 not found: ID does not exist"
Jul 28 12:49:41.467945 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:41.467928    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e5cf3-predictor-567b6c4685-7xcv5"]
Jul 28 12:49:42.252634 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:42.252599    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" path="/var/lib/kubelet/pods/d34136b6-a566-4ef9-9d64-ecb48eda1fd6/volumes"
Jul 28 12:49:42.447138 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:42.447109    2569 generic.go:356] "Generic (PLEG): container finished" podID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerID="22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0" exitCode=0
Jul 28 12:49:42.447269 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:42.447162    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" event={"ID":"12ec7f96-fb1d-45a1-b15b-36d1d129dc71","Type":"ContainerDied","Data":"22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0"}
Jul 28 12:49:43.454266 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:43.454231    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" event={"ID":"12ec7f96-fb1d-45a1-b15b-36d1d129dc71","Type":"ContainerStarted","Data":"d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b"}
Jul 28 12:49:43.454795 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:43.454288    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" event={"ID":"12ec7f96-fb1d-45a1-b15b-36d1d129dc71","Type":"ContainerStarted","Data":"d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a"}
Jul 28 12:49:43.454795 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:43.454608    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:43.454795 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:43.454742    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:43.456105 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:43.456080    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:49:43.476172 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:43.476122    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podStartSLOduration=2.64609267 podStartE2EDuration="6.476107569s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.830014899s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:49:43.474517792 +0000 UTC m=+951.690034463" watchObservedRunningTime="2026-07-28 12:49:43.476107569 +0000 UTC m=+951.691624221"
Jul 28 12:49:44.459461 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:44.459415    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:49:49.464812 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:49.464784    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:49:49.465547 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:49.465517    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:49:59.465523 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:49:59.465484    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:50:09.466067 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:50:09.466019    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:50:19.465691 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:50:19.465612    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:50:29.466295 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:50:29.466255    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:50:39.465493 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:50:39.465451    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:50:49.466447 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:50:49.466410    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:51:27.563249 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.563213    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"]
Jul 28 12:51:27.563708 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.563587    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kube-rbac-proxy"
Jul 28 12:51:27.563708 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.563597    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kube-rbac-proxy"
Jul 28 12:51:27.563708 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.563608    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="storage-initializer"
Jul 28 12:51:27.563708 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.563614    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="storage-initializer"
Jul 28 12:51:27.563708 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.563622    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container"
Jul 28 12:51:27.563708 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.563628    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container"
Jul 28 12:51:27.563708 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.563690    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kube-rbac-proxy"
Jul 28 12:51:27.563708 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.563696    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d34136b6-a566-4ef9-9d64-ecb48eda1fd6" containerName="kserve-container"
Jul 28 12:51:27.567111 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.567080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:27.569325 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.569301    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-75f51-predictor-serving-cert\""
Jul 28 12:51:27.569431 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.569351    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-75f51-kube-rbac-proxy-sar-config\""
Jul 28 12:51:27.578087 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.578064    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"]
Jul 28 12:51:27.594079 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.594047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb5d180-6b49-4019-990e-1aaafb84bf2c-proxy-tls\") pod \"message-dumper-raw-75f51-predictor-ff9697fdd-q9cld\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:27.594211 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.594096    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvxcx\" (UniqueName: \"kubernetes.io/projected/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-api-access-hvxcx\") pod \"message-dumper-raw-75f51-predictor-ff9697fdd-q9cld\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:27.594211 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.594202    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-75f51-predictor-ff9697fdd-q9cld\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:27.662545 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.662503    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"]
Jul 28 12:51:27.662842 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.662792    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container" containerID="cri-o://d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a" gracePeriod=30
Jul 28 12:51:27.662949 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.662848    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kube-rbac-proxy" containerID="cri-o://d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b" gracePeriod=30
Jul 28 12:51:27.694840 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.694810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-75f51-predictor-ff9697fdd-q9cld\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:27.695012 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.694972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb5d180-6b49-4019-990e-1aaafb84bf2c-proxy-tls\") pod \"message-dumper-raw-75f51-predictor-ff9697fdd-q9cld\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:27.695087 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.695040    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvxcx\" (UniqueName: \"kubernetes.io/projected/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-api-access-hvxcx\") pod \"message-dumper-raw-75f51-predictor-ff9697fdd-q9cld\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:27.695191 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:51:27.695171    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-serving-cert: secret "message-dumper-raw-75f51-predictor-serving-cert" not found
Jul 28 12:51:27.695259 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:51:27.695247    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb5d180-6b49-4019-990e-1aaafb84bf2c-proxy-tls podName:abb5d180-6b49-4019-990e-1aaafb84bf2c nodeName:}" failed. No retries permitted until 2026-07-28 12:51:28.195225194 +0000 UTC m=+1056.410741838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/abb5d180-6b49-4019-990e-1aaafb84bf2c-proxy-tls") pod "message-dumper-raw-75f51-predictor-ff9697fdd-q9cld" (UID: "abb5d180-6b49-4019-990e-1aaafb84bf2c") : secret "message-dumper-raw-75f51-predictor-serving-cert" not found
Jul 28 12:51:27.695502 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.695484    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-75f51-predictor-ff9697fdd-q9cld\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:27.705150 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.705130    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvxcx\" (UniqueName: \"kubernetes.io/projected/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-api-access-hvxcx\") pod \"message-dumper-raw-75f51-predictor-ff9697fdd-q9cld\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:27.937401 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.937317    2569 generic.go:356] "Generic (PLEG): container finished" podID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerID="d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b" exitCode=2
Jul 28 12:51:27.937401 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:27.937378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" event={"ID":"12ec7f96-fb1d-45a1-b15b-36d1d129dc71","Type":"ContainerDied","Data":"d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b"}
Jul 28 12:51:28.197978 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:28.197881    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb5d180-6b49-4019-990e-1aaafb84bf2c-proxy-tls\") pod \"message-dumper-raw-75f51-predictor-ff9697fdd-q9cld\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:28.200247 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:28.200217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb5d180-6b49-4019-990e-1aaafb84bf2c-proxy-tls\") pod \"message-dumper-raw-75f51-predictor-ff9697fdd-q9cld\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") " pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:28.478144 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:28.478103    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:28.609389 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:28.609364    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"]
Jul 28 12:51:28.611605 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:51:28.611577    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb5d180_6b49_4019_990e_1aaafb84bf2c.slice/crio-6b5b882d884644c902f31fa81a9bfe43195114616814fe3dbfbf250c10586081 WatchSource:0}: Error finding container 6b5b882d884644c902f31fa81a9bfe43195114616814fe3dbfbf250c10586081: Status 404 returned error can't find the container with id 6b5b882d884644c902f31fa81a9bfe43195114616814fe3dbfbf250c10586081
Jul 28 12:51:28.943398 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:28.943309    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld" event={"ID":"abb5d180-6b49-4019-990e-1aaafb84bf2c","Type":"ContainerStarted","Data":"6b5b882d884644c902f31fa81a9bfe43195114616814fe3dbfbf250c10586081"}
Jul 28 12:51:29.459764 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:29.459725    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 12:51:29.466169 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:29.466136    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:51:30.954719 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:30.954678    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld" event={"ID":"abb5d180-6b49-4019-990e-1aaafb84bf2c","Type":"ContainerStarted","Data":"ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6"}
Jul 28 12:51:30.955165 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:30.954741    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld" event={"ID":"abb5d180-6b49-4019-990e-1aaafb84bf2c","Type":"ContainerStarted","Data":"21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93"}
Jul 28 12:51:30.955165 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:30.954893    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:30.955165 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:30.954942    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:30.956843 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:30.956820    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:51:30.975741 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:30.975689    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld" podStartSLOduration=2.602866702 podStartE2EDuration="3.975672187s" totalImagesPullingTime="1.372805485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:51:30.973690828 +0000 UTC m=+1059.189207478" watchObservedRunningTime="2026-07-28 12:51:30.975672187 +0000 UTC m=+1059.191188839"
Jul 28 12:51:31.323282 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.323252    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:51:31.427688 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.427634    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kserve-provision-location\") pod \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") "
Jul 28 12:51:31.427880 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.427711    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-rbac-proxy-sar-config\") pod \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") "
Jul 28 12:51:31.427880 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.427771    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-cabundle-cert\") pod \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") "
Jul 28 12:51:31.427880 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.427809    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpq74\" (UniqueName: \"kubernetes.io/projected/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-api-access-jpq74\") pod \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") "
Jul 28 12:51:31.427880 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.427858    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-proxy-tls\") pod \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\" (UID: \"12ec7f96-fb1d-45a1-b15b-36d1d129dc71\") "
Jul 28 12:51:31.428128 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.428013    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12ec7f96-fb1d-45a1-b15b-36d1d129dc71" (UID: "12ec7f96-fb1d-45a1-b15b-36d1d129dc71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:51:31.428192 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.428156    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12ec7f96-fb1d-45a1-b15b-36d1d129dc71" (UID: "12ec7f96-fb1d-45a1-b15b-36d1d129dc71"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:51:31.428249 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.428219    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kserve-provision-location\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:51:31.428249 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.428232    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:51:31.428333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.428263    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12ec7f96-fb1d-45a1-b15b-36d1d129dc71" (UID: "12ec7f96-fb1d-45a1-b15b-36d1d129dc71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:51:31.430101 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.430075    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12ec7f96-fb1d-45a1-b15b-36d1d129dc71" (UID: "12ec7f96-fb1d-45a1-b15b-36d1d129dc71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:51:31.430101 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.430085    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-api-access-jpq74" (OuterVolumeSpecName: "kube-api-access-jpq74") pod "12ec7f96-fb1d-45a1-b15b-36d1d129dc71" (UID: "12ec7f96-fb1d-45a1-b15b-36d1d129dc71"). InnerVolumeSpecName "kube-api-access-jpq74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:51:31.528964 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.528859    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-cabundle-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:51:31.528964 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.528908    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpq74\" (UniqueName: \"kubernetes.io/projected/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-kube-api-access-jpq74\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:51:31.528964 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.528925    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12ec7f96-fb1d-45a1-b15b-36d1d129dc71-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:51:31.960886 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.960847    2569 generic.go:356] "Generic (PLEG): container finished" podID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerID="d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a" exitCode=0
Jul 28 12:51:31.961397 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.960925    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"
Jul 28 12:51:31.961397 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.960933    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" event={"ID":"12ec7f96-fb1d-45a1-b15b-36d1d129dc71","Type":"ContainerDied","Data":"d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a"}
Jul 28 12:51:31.961397 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.961076    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d" event={"ID":"12ec7f96-fb1d-45a1-b15b-36d1d129dc71","Type":"ContainerDied","Data":"7cde39cf3ad7e864c0c5f6e6c6ef19d67a9500542b63a2edff4a236ac4eb0c9c"}
Jul 28 12:51:31.961397 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.961120    2569 scope.go:117] "RemoveContainer" containerID="d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b"
Jul 28 12:51:31.970052 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.970033    2569 scope.go:117] "RemoveContainer" containerID="d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a"
Jul 28 12:51:31.977486 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.977461    2569 scope.go:117] "RemoveContainer" containerID="22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0"
Jul 28 12:51:31.987356 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.987322    2569 scope.go:117] "RemoveContainer" containerID="d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b"
Jul 28 12:51:31.987769 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:51:31.987740    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b\": container with ID starting with d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b not found: ID does not exist" containerID="d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b"
Jul 28 12:51:31.987926 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.987780    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b"} err="failed to get container status \"d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b\": rpc error: code = NotFound desc = could not find container \"d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b\": container with ID starting with d531e688000c11547b92193770a3240e32d2ffa99869fdcdf1d52e8908289c2b not found: ID does not exist"
Jul 28 12:51:31.987926 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.987919    2569 scope.go:117] "RemoveContainer" containerID="d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a"
Jul 28 12:51:31.988356 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:51:31.988326    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a\": container with ID starting with d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a not found: ID does not exist" containerID="d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a"
Jul 28 12:51:31.988463 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.988362    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a"} err="failed to get container status \"d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a\": rpc error: code = NotFound desc = could not find container \"d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a\": container with ID starting with d3030db65aff980a3b0ffd468983070f60b4d1838cc7b29a919c3c43fa3c641a not found: ID does not exist"
Jul 28 12:51:31.988463 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.988385    2569 scope.go:117] "RemoveContainer" containerID="22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0"
Jul 28 12:51:31.988574 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.988546    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"]
Jul 28 12:51:31.988715 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:51:31.988676    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0\": container with ID starting with 22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0 not found: ID does not exist" containerID="22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0"
Jul 28 12:51:31.988778 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.988720    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0"} err="failed to get container status \"22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0\": rpc error: code = NotFound desc = could not find container \"22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0\": container with ID starting with 22e4681ebbc42042da4df3b3156ed70b7cbb87fccdd6f21a87e03b971b74cdd0 not found: ID does not exist"
Jul 28 12:51:31.990360 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:31.990339    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f8c4-predictor-cdbf9b77c-5ls5d"]
Jul 28 12:51:32.253141 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:32.253063    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" path="/var/lib/kubelet/pods/12ec7f96-fb1d-45a1-b15b-36d1d129dc71/volumes"
Jul 28 12:51:37.972013 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:51:37.971956    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:53:22.658640 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:22.658566    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-75f51-predictor-ff9697fdd-q9cld_abb5d180-6b49-4019-990e-1aaafb84bf2c/kserve-container/0.log"
Jul 28 12:53:23.013409 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.013376    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"]
Jul 28 12:53:23.013626 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.013602    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld" podUID="abb5d180-6b49-4019-990e-1aaafb84bf2c" containerName="kserve-container" containerID="cri-o://21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93" gracePeriod=30
Jul 28 12:53:23.013691 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.013635    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld" podUID="abb5d180-6b49-4019-990e-1aaafb84bf2c" containerName="kube-rbac-proxy" containerID="cri-o://ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6" gracePeriod=30
Jul 28 12:53:23.261139 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.261114    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:53:23.422830 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.422734    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-rbac-proxy-sar-config\") pod \"abb5d180-6b49-4019-990e-1aaafb84bf2c\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") "
Jul 28 12:53:23.422830 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.422791    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvxcx\" (UniqueName: \"kubernetes.io/projected/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-api-access-hvxcx\") pod \"abb5d180-6b49-4019-990e-1aaafb84bf2c\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") "
Jul 28 12:53:23.422830 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.422837    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb5d180-6b49-4019-990e-1aaafb84bf2c-proxy-tls\") pod \"abb5d180-6b49-4019-990e-1aaafb84bf2c\" (UID: \"abb5d180-6b49-4019-990e-1aaafb84bf2c\") "
Jul 28 12:53:23.423174 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.423149    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abb5d180-6b49-4019-990e-1aaafb84bf2c" (UID: "abb5d180-6b49-4019-990e-1aaafb84bf2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:53:23.424880 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.424853    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-api-access-hvxcx" (OuterVolumeSpecName: "kube-api-access-hvxcx") pod "abb5d180-6b49-4019-990e-1aaafb84bf2c" (UID: "abb5d180-6b49-4019-990e-1aaafb84bf2c"). InnerVolumeSpecName "kube-api-access-hvxcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:53:23.425014 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.424965    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb5d180-6b49-4019-990e-1aaafb84bf2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abb5d180-6b49-4019-990e-1aaafb84bf2c" (UID: "abb5d180-6b49-4019-990e-1aaafb84bf2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:53:23.463194 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.463164    2569 generic.go:356] "Generic (PLEG): container finished" podID="abb5d180-6b49-4019-990e-1aaafb84bf2c" containerID="ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6" exitCode=2
Jul 28 12:53:23.463194 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.463188    2569 generic.go:356] "Generic (PLEG): container finished" podID="abb5d180-6b49-4019-990e-1aaafb84bf2c" containerID="21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93" exitCode=2
Jul 28 12:53:23.463339 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.463208    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld" event={"ID":"abb5d180-6b49-4019-990e-1aaafb84bf2c","Type":"ContainerDied","Data":"ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6"}
Jul 28 12:53:23.463339 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.463211    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"
Jul 28 12:53:23.463339 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.463256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld" event={"ID":"abb5d180-6b49-4019-990e-1aaafb84bf2c","Type":"ContainerDied","Data":"21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93"}
Jul 28 12:53:23.463339 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.463277    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld" event={"ID":"abb5d180-6b49-4019-990e-1aaafb84bf2c","Type":"ContainerDied","Data":"6b5b882d884644c902f31fa81a9bfe43195114616814fe3dbfbf250c10586081"}
Jul 28 12:53:23.463339 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.463299    2569 scope.go:117] "RemoveContainer" containerID="ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6"
Jul 28 12:53:23.471579 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.471562    2569 scope.go:117] "RemoveContainer" containerID="21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93"
Jul 28 12:53:23.479195 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.479179    2569 scope.go:117] "RemoveContainer" containerID="ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6"
Jul 28 12:53:23.479407 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:53:23.479387    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6\": container with ID starting with ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6 not found: ID does not exist" containerID="ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6"
Jul 28 12:53:23.479483 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.479420    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6"} err="failed to get container status \"ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6\": rpc error: code = NotFound desc = could not find container \"ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6\": container with ID starting with ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6 not found: ID does not exist"
Jul 28 12:53:23.479483 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.479444    2569 scope.go:117] "RemoveContainer" containerID="21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93"
Jul 28 12:53:23.479707 ip-10-0-142-139 kubenswrapper[2569]: E0728 12:53:23.479689    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93\": container with ID starting with 21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93 not found: ID does not exist" containerID="21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93"
Jul 28 12:53:23.479773 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.479714    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93"} err="failed to get container status \"21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93\": rpc error: code = NotFound desc = could not find container \"21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93\": container with ID starting with 21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93 not found: ID does not exist"
Jul 28 12:53:23.479773 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.479732    2569 scope.go:117] "RemoveContainer" containerID="ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6"
Jul 28 12:53:23.479910 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.479889    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6"} err="failed to get container status \"ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6\": rpc error: code = NotFound desc = could not find container \"ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6\": container with ID starting with ee90799549a019130e17bde5c03fe9b47862c27e25e48e7a13cdb8669029d5b6 not found: ID does not exist"
Jul 28 12:53:23.479962 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.479910    2569 scope.go:117] "RemoveContainer" containerID="21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93"
Jul 28 12:53:23.480123 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.480108    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93"} err="failed to get container status \"21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93\": rpc error: code = NotFound desc = could not find container \"21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93\": container with ID starting with 21d62937ae511d2e0d10f7c95eda767b6e897f38b45fd94e13c2b31750dbda93 not found: ID does not exist"
Jul 28 12:53:23.491711 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.491685    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"]
Jul 28 12:53:23.499010 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.498977    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-75f51-predictor-ff9697fdd-q9cld"]
Jul 28 12:53:23.524113 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.524087    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb5d180-6b49-4019-990e-1aaafb84bf2c-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:53:23.524113 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.524112    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:53:23.524244 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:23.524126    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvxcx\" (UniqueName: \"kubernetes.io/projected/abb5d180-6b49-4019-990e-1aaafb84bf2c-kube-api-access-hvxcx\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 12:53:24.252393 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:53:24.252356    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abb5d180-6b49-4019-990e-1aaafb84bf2c" path="/var/lib/kubelet/pods/abb5d180-6b49-4019-990e-1aaafb84bf2c/volumes"
Jul 28 12:59:23.780937 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.780856    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"]
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781231    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="storage-initializer"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781243    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="storage-initializer"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781255    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kube-rbac-proxy"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781260    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kube-rbac-proxy"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781273    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb5d180-6b49-4019-990e-1aaafb84bf2c" containerName="kube-rbac-proxy"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781279    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb5d180-6b49-4019-990e-1aaafb84bf2c" containerName="kube-rbac-proxy"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781286    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb5d180-6b49-4019-990e-1aaafb84bf2c" containerName="kserve-container"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781291    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb5d180-6b49-4019-990e-1aaafb84bf2c" containerName="kserve-container"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781302    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781309    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781366    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kserve-container"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781379    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb5d180-6b49-4019-990e-1aaafb84bf2c" containerName="kube-rbac-proxy"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781387    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb5d180-6b49-4019-990e-1aaafb84bf2c" containerName="kserve-container"
Jul 28 12:59:23.781413 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.781395    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ec7f96-fb1d-45a1-b15b-36d1d129dc71" containerName="kube-rbac-proxy"
Jul 28 12:59:23.784487 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.784470    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:23.787075 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.787053    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 12:59:23.787075 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.787068    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 12:59:23.787075 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.787052    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 12:59:23.787325 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.787055    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 12:59:23.787883 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.787868    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 12:59:23.787883 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.787876    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 12:59:23.788024 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.787941    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rtw6d\""
Jul 28 12:59:23.794815 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.794791    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"]
Jul 28 12:59:23.909733 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.909697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:23.909733 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.909735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:23.909933 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.909762    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nqcb\" (UniqueName: \"kubernetes.io/projected/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-api-access-6nqcb\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:23.909933 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.909876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:23.909933 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:23.909912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.010926 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.010893    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.010926 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.010931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.011140 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.010973    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.011140 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.011007    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.011140 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.011029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nqcb\" (UniqueName: \"kubernetes.io/projected/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-api-access-6nqcb\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.011482 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.011447    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.011737 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.011718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.011809 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.011792    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.013333 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.013314    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.018709 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.018689    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nqcb\" (UniqueName: \"kubernetes.io/projected/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-api-access-6nqcb\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-6jmh6\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.094485 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.094403    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:24.224875 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:24.224852    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"]
Jul 28 12:59:24.226831 ip-10-0-142-139 kubenswrapper[2569]: W0728 12:59:24.226798    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45217a6_e017_48dc_9cd6_c84fce9f1c2e.slice/crio-d648ffba9e66efc8871335da000c757c2a67c0e86538642a373a51074253889c WatchSource:0}: Error finding container d648ffba9e66efc8871335da000c757c2a67c0e86538642a373a51074253889c: Status 404 returned error can't find the container with id d648ffba9e66efc8871335da000c757c2a67c0e86538642a373a51074253889c
Jul 28 12:59:25.073189 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:25.073157    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" event={"ID":"a45217a6-e017-48dc-9cd6-c84fce9f1c2e","Type":"ContainerStarted","Data":"0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a"}
Jul 28 12:59:25.073538 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:25.073213    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" event={"ID":"a45217a6-e017-48dc-9cd6-c84fce9f1c2e","Type":"ContainerStarted","Data":"d648ffba9e66efc8871335da000c757c2a67c0e86538642a373a51074253889c"}
Jul 28 12:59:29.092412 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:29.092377    2569 generic.go:356] "Generic (PLEG): container finished" podID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerID="0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a" exitCode=0
Jul 28 12:59:29.092764 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:29.092452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" event={"ID":"a45217a6-e017-48dc-9cd6-c84fce9f1c2e","Type":"ContainerDied","Data":"0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a"}
Jul 28 12:59:29.093553 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:29.093516    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:59:43.172664 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:43.172630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" event={"ID":"a45217a6-e017-48dc-9cd6-c84fce9f1c2e","Type":"ContainerStarted","Data":"e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319"}
Jul 28 12:59:43.173043 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:43.172691    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" event={"ID":"a45217a6-e017-48dc-9cd6-c84fce9f1c2e","Type":"ContainerStarted","Data":"5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612"}
Jul 28 12:59:43.173043 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:43.172887    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:43.223454 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:43.223398    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podStartSLOduration=3.239477067 podStartE2EDuration="20.223380085s" totalImagesPullingTime="13.088173506s" totalInitContainerRuntime="3.895729512s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:59:43.222581401 +0000 UTC m=+1551.438098054" watchObservedRunningTime="2026-07-28 12:59:43.223380085 +0000 UTC m=+1551.438896741"
Jul 28 12:59:44.177805 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:44.177774    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:44.179136 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:44.179110    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 12:59:45.183174 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:45.183130    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 12:59:50.188845 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:50.188816    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 12:59:50.189426 ip-10-0-142-139 kubenswrapper[2569]: I0728 12:59:50.189398    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 13:00:00.189557 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:00:00.189518    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 13:00:10.189854 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:00:10.189804    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 13:00:20.189680 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:00:20.189640    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 13:00:30.190248 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:00:30.190205    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 13:00:40.190078 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:00:40.190033    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 13:00:50.189494 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:00:50.189401    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 13:01:00.190747 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:00.190715    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 13:01:03.980064 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:03.980031    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"]
Jul 28 13:01:03.980472 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:03.980268    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container" containerID="cri-o://5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612" gracePeriod=30
Jul 28 13:01:03.980472 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:03.980297    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kube-rbac-proxy" containerID="cri-o://e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319" gracePeriod=30
Jul 28 13:01:04.540132 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:04.540092    2569 generic.go:356] "Generic (PLEG): container finished" podID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerID="e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319" exitCode=2
Jul 28 13:01:04.540295 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:04.540166    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" event={"ID":"a45217a6-e017-48dc-9cd6-c84fce9f1c2e","Type":"ContainerDied","Data":"e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319"}
Jul 28 13:01:05.183402 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:05.183358    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 13:01:08.533237 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.533213    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 13:01:08.560457 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.560423    2569 generic.go:356] "Generic (PLEG): container finished" podID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerID="5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612" exitCode=0
Jul 28 13:01:08.560640 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.560468    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" event={"ID":"a45217a6-e017-48dc-9cd6-c84fce9f1c2e","Type":"ContainerDied","Data":"5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612"}
Jul 28 13:01:08.560640 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.560500    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"
Jul 28 13:01:08.560640 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.560531    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6" event={"ID":"a45217a6-e017-48dc-9cd6-c84fce9f1c2e","Type":"ContainerDied","Data":"d648ffba9e66efc8871335da000c757c2a67c0e86538642a373a51074253889c"}
Jul 28 13:01:08.560640 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.560568    2569 scope.go:117] "RemoveContainer" containerID="e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319"
Jul 28 13:01:08.571378 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.571348    2569 scope.go:117] "RemoveContainer" containerID="5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612"
Jul 28 13:01:08.579724 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.579707    2569 scope.go:117] "RemoveContainer" containerID="0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a"
Jul 28 13:01:08.587800 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.587782    2569 scope.go:117] "RemoveContainer" containerID="e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319"
Jul 28 13:01:08.588185 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:01:08.588164    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319\": container with ID starting with e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319 not found: ID does not exist" containerID="e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319"
Jul 28 13:01:08.588269 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.588194    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319"} err="failed to get container status \"e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319\": rpc error: code = NotFound desc = could not find container \"e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319\": container with ID starting with e7e0f26886d49893e0f21803d6668a1214d1272c23f5b2b121e4cfcdee078319 not found: ID does not exist"
Jul 28 13:01:08.588269 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.588217    2569 scope.go:117] "RemoveContainer" containerID="5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612"
Jul 28 13:01:08.588470 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:01:08.588445    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612\": container with ID starting with 5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612 not found: ID does not exist" containerID="5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612"
Jul 28 13:01:08.588514 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.588477    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612"} err="failed to get container status \"5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612\": rpc error: code = NotFound desc = could not find container \"5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612\": container with ID starting with 5b2e7ba35c1a5e3aaf0635725beb48398dba136de80d8213a855c8fdd3a74612 not found: ID does not exist"
Jul 28 13:01:08.588514 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.588494    2569 scope.go:117] "RemoveContainer" containerID="0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a"
Jul 28 13:01:08.588714 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:01:08.588693    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a\": container with ID starting with 0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a not found: ID does not exist" containerID="0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a"
Jul 28 13:01:08.588755 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.588722    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a"} err="failed to get container status \"0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a\": rpc error: code = NotFound desc = could not find container \"0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a\": container with ID starting with 0a804794e5872217d2fc390916d3dd987672219ef1544749b4a65bc35b8f308a not found: ID does not exist"
Jul 28 13:01:08.655311 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.655228    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kserve-provision-location\") pod \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") "
Jul 28 13:01:08.655311 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.655291    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-cabundle-cert\") pod \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") "
Jul 28 13:01:08.655311 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.655315    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-proxy-tls\") pod \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") "
Jul 28 13:01:08.655528 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.655363    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-rbac-proxy-sar-config\") pod \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") "
Jul 28 13:01:08.655528 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.655402    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nqcb\" (UniqueName: \"kubernetes.io/projected/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-api-access-6nqcb\") pod \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\" (UID: \"a45217a6-e017-48dc-9cd6-c84fce9f1c2e\") "
Jul 28 13:01:08.655642 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.655613    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a45217a6-e017-48dc-9cd6-c84fce9f1c2e" (UID: "a45217a6-e017-48dc-9cd6-c84fce9f1c2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:01:08.655750 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.655720    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a45217a6-e017-48dc-9cd6-c84fce9f1c2e" (UID: "a45217a6-e017-48dc-9cd6-c84fce9f1c2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:08.655750 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.655737    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a45217a6-e017-48dc-9cd6-c84fce9f1c2e" (UID: "a45217a6-e017-48dc-9cd6-c84fce9f1c2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:08.657487 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.657465    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a45217a6-e017-48dc-9cd6-c84fce9f1c2e" (UID: "a45217a6-e017-48dc-9cd6-c84fce9f1c2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:01:08.657670 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.657650    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-api-access-6nqcb" (OuterVolumeSpecName: "kube-api-access-6nqcb") pod "a45217a6-e017-48dc-9cd6-c84fce9f1c2e" (UID: "a45217a6-e017-48dc-9cd6-c84fce9f1c2e"). InnerVolumeSpecName "kube-api-access-6nqcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:01:08.756323 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.756286    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:01:08.756323 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.756318    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nqcb\" (UniqueName: \"kubernetes.io/projected/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kube-api-access-6nqcb\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:01:08.756323 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.756330    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-kserve-provision-location\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:01:08.756580 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.756340    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-cabundle-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:01:08.756580 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.756349    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a45217a6-e017-48dc-9cd6-c84fce9f1c2e-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:01:08.885731 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.885703    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"]
Jul 28 13:01:08.888514 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:08.888485    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-6jmh6"]
Jul 28 13:01:09.141430 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.141394    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"]
Jul 28 13:01:09.141758 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.141746    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container"
Jul 28 13:01:09.141810 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.141759    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container"
Jul 28 13:01:09.141810 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.141771    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="storage-initializer"
Jul 28 13:01:09.141810 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.141777    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="storage-initializer"
Jul 28 13:01:09.141810 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.141791    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kube-rbac-proxy"
Jul 28 13:01:09.141810 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.141798    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kube-rbac-proxy"
Jul 28 13:01:09.141998 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.141848    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kserve-container"
Jul 28 13:01:09.141998 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.141859    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" containerName="kube-rbac-proxy"
Jul 28 13:01:09.146857 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.146831    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.148274 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.148252    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"]
Jul 28 13:01:09.149891 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.149873    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:01:09.150014 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.149907    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rtw6d\""
Jul 28 13:01:09.150134 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.150120    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 13:01:09.150205 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.150173    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 13:01:09.150289 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.150273    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:01:09.150371 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.150355    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:01:09.150446 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.150426    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:01:09.151684 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.151670    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.153884 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.153866    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 13:01:09.164060 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.164040    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"]
Jul 28 13:01:09.167481 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.167458    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"]
Jul 28 13:01:09.260271 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.260235    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89efd3d2-ceb4-4200-8b11-24b899ce188a-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.260271 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.260270    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.260506 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.260292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.260506 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.260310    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cdf12ee-7c10-4638-bece-8fa0305dec07-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.260506 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.260398    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.260506 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.260447    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxjp2\" (UniqueName: \"kubernetes.io/projected/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-api-access-lxjp2\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.260506 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.260473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89efd3d2-ceb4-4200-8b11-24b899ce188a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.260506 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.260500    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.260766 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.260524    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cdf12ee-7c10-4638-bece-8fa0305dec07-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.260766 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.260571    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzvnn\" (UniqueName: \"kubernetes.io/projected/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-api-access-wzvnn\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.361801 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.361763    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.361801 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.361802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cdf12ee-7c10-4638-bece-8fa0305dec07-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.362092 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.361820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzvnn\" (UniqueName: \"kubernetes.io/projected/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-api-access-wzvnn\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.362092 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.361858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89efd3d2-ceb4-4200-8b11-24b899ce188a-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.362092 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.361878    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.362092 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.361904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.362092 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.361933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cdf12ee-7c10-4638-bece-8fa0305dec07-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.362092 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:01:09.361953    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 13:01:09.362092 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.362024    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.362092 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:01:09.362050    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdf12ee-7c10-4638-bece-8fa0305dec07-proxy-tls podName:2cdf12ee-7c10-4638-bece-8fa0305dec07 nodeName:}" failed. No retries permitted until 2026-07-28 13:01:09.862030099 +0000 UTC m=+1638.077546746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2cdf12ee-7c10-4638-bece-8fa0305dec07-proxy-tls") pod "isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" (UID: "2cdf12ee-7c10-4638-bece-8fa0305dec07") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 13:01:09.362517 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.362120    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxjp2\" (UniqueName: \"kubernetes.io/projected/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-api-access-lxjp2\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.362517 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.362156    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89efd3d2-ceb4-4200-8b11-24b899ce188a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.362633 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.362517    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cdf12ee-7c10-4638-bece-8fa0305dec07-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.362692 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.362672    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.362758 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.362737    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.362917 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.362897    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89efd3d2-ceb4-4200-8b11-24b899ce188a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.362955 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.362923    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.363173 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.363150    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.364511 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.364490    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89efd3d2-ceb4-4200-8b11-24b899ce188a-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.370691 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.370672    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxjp2\" (UniqueName: \"kubernetes.io/projected/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-api-access-lxjp2\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.370772 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.370736    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzvnn\" (UniqueName: \"kubernetes.io/projected/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-api-access-wzvnn\") pod \"isvc-canary-rollback-predictor-ff46f797c-c6xvk\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.458888 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.458865    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:09.590323 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.590298    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"]
Jul 28 13:01:09.592426 ip-10-0-142-139 kubenswrapper[2569]: W0728 13:01:09.592402    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89efd3d2_ceb4_4200_8b11_24b899ce188a.slice/crio-27e4c8a2c02aee13d158620f6dcecd1b93b7a886583124ec067d4124b5032044 WatchSource:0}: Error finding container 27e4c8a2c02aee13d158620f6dcecd1b93b7a886583124ec067d4124b5032044: Status 404 returned error can't find the container with id 27e4c8a2c02aee13d158620f6dcecd1b93b7a886583124ec067d4124b5032044
Jul 28 13:01:09.866338 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.866252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cdf12ee-7c10-4638-bece-8fa0305dec07-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:09.868588 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:09.868563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cdf12ee-7c10-4638-bece-8fa0305dec07-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:10.066048 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:10.066018    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:10.202821 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:10.202788    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"]
Jul 28 13:01:10.206293 ip-10-0-142-139 kubenswrapper[2569]: W0728 13:01:10.206266    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdf12ee_7c10_4638_bece_8fa0305dec07.slice/crio-fae5356106af5643ae1f6e8a8cad5b58024ef9498e312ebe20097fb90fa064c6 WatchSource:0}: Error finding container fae5356106af5643ae1f6e8a8cad5b58024ef9498e312ebe20097fb90fa064c6: Status 404 returned error can't find the container with id fae5356106af5643ae1f6e8a8cad5b58024ef9498e312ebe20097fb90fa064c6
Jul 28 13:01:10.252624 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:10.252597    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a45217a6-e017-48dc-9cd6-c84fce9f1c2e" path="/var/lib/kubelet/pods/a45217a6-e017-48dc-9cd6-c84fce9f1c2e/volumes"
Jul 28 13:01:10.575731 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:10.575696    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" event={"ID":"2cdf12ee-7c10-4638-bece-8fa0305dec07","Type":"ContainerStarted","Data":"2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042"}
Jul 28 13:01:10.575912 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:10.575766    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" event={"ID":"2cdf12ee-7c10-4638-bece-8fa0305dec07","Type":"ContainerStarted","Data":"fae5356106af5643ae1f6e8a8cad5b58024ef9498e312ebe20097fb90fa064c6"}
Jul 28 13:01:10.577314 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:10.577286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" event={"ID":"89efd3d2-ceb4-4200-8b11-24b899ce188a","Type":"ContainerStarted","Data":"e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b"}
Jul 28 13:01:10.577425 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:10.577341    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" event={"ID":"89efd3d2-ceb4-4200-8b11-24b899ce188a","Type":"ContainerStarted","Data":"27e4c8a2c02aee13d158620f6dcecd1b93b7a886583124ec067d4124b5032044"}
Jul 28 13:01:13.595559 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:13.595457    2569 generic.go:356] "Generic (PLEG): container finished" podID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerID="e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b" exitCode=0
Jul 28 13:01:13.595559 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:13.595530    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" event={"ID":"89efd3d2-ceb4-4200-8b11-24b899ce188a","Type":"ContainerDied","Data":"e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b"}
Jul 28 13:01:14.602752 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:14.602713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" event={"ID":"89efd3d2-ceb4-4200-8b11-24b899ce188a","Type":"ContainerStarted","Data":"af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df"}
Jul 28 13:01:14.603227 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:14.602779    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" event={"ID":"89efd3d2-ceb4-4200-8b11-24b899ce188a","Type":"ContainerStarted","Data":"f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6"}
Jul 28 13:01:14.603227 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:14.603119    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:14.603227 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:14.603184    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:14.604289 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:14.604257    2569 generic.go:356] "Generic (PLEG): container finished" podID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerID="2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042" exitCode=0
Jul 28 13:01:14.604400 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:14.604327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" event={"ID":"2cdf12ee-7c10-4638-bece-8fa0305dec07","Type":"ContainerDied","Data":"2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042"}
Jul 28 13:01:14.605068 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:14.605041    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:01:14.623450 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:14.623405    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podStartSLOduration=1.973438476 podStartE2EDuration="5.623385312s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.649946836s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:01:14.620760816 +0000 UTC m=+1642.836277469" watchObservedRunningTime="2026-07-28 13:01:14.623385312 +0000 UTC m=+1642.838901965"
Jul 28 13:01:15.611202 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:15.611165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" event={"ID":"2cdf12ee-7c10-4638-bece-8fa0305dec07","Type":"ContainerStarted","Data":"082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd"}
Jul 28 13:01:15.611603 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:15.611236    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" event={"ID":"2cdf12ee-7c10-4638-bece-8fa0305dec07","Type":"ContainerStarted","Data":"53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a"}
Jul 28 13:01:15.611603 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:15.611501    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:01:15.611737 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:15.611718    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:15.632774 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:15.632716    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podStartSLOduration=2.942213025 podStartE2EDuration="6.632700192s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690487167s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:01:15.630455559 +0000 UTC m=+1643.845972210" watchObservedRunningTime="2026-07-28 13:01:15.632700192 +0000 UTC m=+1643.848216844"
Jul 28 13:01:16.615946 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:16.615917    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:16.617306 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:16.617280    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:01:17.619978 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:17.619940    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:01:20.615572 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:20.615535    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:01:20.616070 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:20.616043    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:01:22.624564 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:22.624533    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:01:22.625059 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:22.625027    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:01:30.615980 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:30.615940    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:01:32.625599 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:32.625562    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:01:40.616511 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:40.616473    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:01:42.625355 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:42.625317    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:01:50.616354 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:50.616318    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:01:52.625856 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:01:52.625770    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:02:00.616065 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:00.616020    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:02:02.625013 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:02.624955    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:02:10.616530 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:10.616488    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:02:12.625846 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:12.625812    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:02:20.616768 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:20.616683    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:02:22.625111 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:22.625074    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:02:30.617082 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:30.617051    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:02:32.626776 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:32.626750    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:02:39.186597 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:39.186561    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"]
Jul 28 13:02:39.187075 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:39.186794    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" containerID="cri-o://53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a" gracePeriod=30
Jul 28 13:02:39.187075 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:39.186839    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kube-rbac-proxy" containerID="cri-o://082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd" gracePeriod=30
Jul 28 13:02:40.013064 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:40.013034    2569 generic.go:356] "Generic (PLEG): container finished" podID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerID="082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd" exitCode=2
Jul 28 13:02:40.013243 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:40.013115    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" event={"ID":"2cdf12ee-7c10-4638-bece-8fa0305dec07","Type":"ContainerDied","Data":"082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd"}
Jul 28 13:02:42.620716 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:42.620678    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 13:02:42.625077 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:42.625052    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:02:43.832120 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.832094    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:02:43.914680 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.914600    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cdf12ee-7c10-4638-bece-8fa0305dec07-proxy-tls\") pod \"2cdf12ee-7c10-4638-bece-8fa0305dec07\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") "
Jul 28 13:02:43.914680 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.914637    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-rbac-proxy-sar-config\") pod \"2cdf12ee-7c10-4638-bece-8fa0305dec07\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") "
Jul 28 13:02:43.914863 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.914682    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cdf12ee-7c10-4638-bece-8fa0305dec07-kserve-provision-location\") pod \"2cdf12ee-7c10-4638-bece-8fa0305dec07\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") "
Jul 28 13:02:43.914863 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.914723    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-cabundle-cert\") pod \"2cdf12ee-7c10-4638-bece-8fa0305dec07\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") "
Jul 28 13:02:43.914863 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.914776    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxjp2\" (UniqueName: \"kubernetes.io/projected/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-api-access-lxjp2\") pod \"2cdf12ee-7c10-4638-bece-8fa0305dec07\" (UID: \"2cdf12ee-7c10-4638-bece-8fa0305dec07\") "
Jul 28 13:02:43.915081 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.915060    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdf12ee-7c10-4638-bece-8fa0305dec07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2cdf12ee-7c10-4638-bece-8fa0305dec07" (UID: "2cdf12ee-7c10-4638-bece-8fa0305dec07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:02:43.915147 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.915104    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2cdf12ee-7c10-4638-bece-8fa0305dec07" (UID: "2cdf12ee-7c10-4638-bece-8fa0305dec07"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:02:43.915147 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.915130    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2cdf12ee-7c10-4638-bece-8fa0305dec07" (UID: "2cdf12ee-7c10-4638-bece-8fa0305dec07"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:02:43.916751 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.916716    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cdf12ee-7c10-4638-bece-8fa0305dec07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2cdf12ee-7c10-4638-bece-8fa0305dec07" (UID: "2cdf12ee-7c10-4638-bece-8fa0305dec07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:02:43.916864 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:43.916756    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-api-access-lxjp2" (OuterVolumeSpecName: "kube-api-access-lxjp2") pod "2cdf12ee-7c10-4638-bece-8fa0305dec07" (UID: "2cdf12ee-7c10-4638-bece-8fa0305dec07"). InnerVolumeSpecName "kube-api-access-lxjp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:02:44.016367 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.016333    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cdf12ee-7c10-4638-bece-8fa0305dec07-kserve-provision-location\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:02:44.016367 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.016363    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-cabundle-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:02:44.016367 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.016374    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxjp2\" (UniqueName: \"kubernetes.io/projected/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-api-access-lxjp2\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:02:44.016596 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.016387    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cdf12ee-7c10-4638-bece-8fa0305dec07-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:02:44.016596 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.016395    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cdf12ee-7c10-4638-bece-8fa0305dec07-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:02:44.032731 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.032697    2569 generic.go:356] "Generic (PLEG): container finished" podID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerID="53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a" exitCode=0
Jul 28 13:02:44.032841 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.032754    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"
Jul 28 13:02:44.032841 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.032783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" event={"ID":"2cdf12ee-7c10-4638-bece-8fa0305dec07","Type":"ContainerDied","Data":"53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a"}
Jul 28 13:02:44.032934 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.032843    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs" event={"ID":"2cdf12ee-7c10-4638-bece-8fa0305dec07","Type":"ContainerDied","Data":"fae5356106af5643ae1f6e8a8cad5b58024ef9498e312ebe20097fb90fa064c6"}
Jul 28 13:02:44.032934 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.032871    2569 scope.go:117] "RemoveContainer" containerID="082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd"
Jul 28 13:02:44.041759 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.041741    2569 scope.go:117] "RemoveContainer" containerID="53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a"
Jul 28 13:02:44.049119 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.049103    2569 scope.go:117] "RemoveContainer" containerID="2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042"
Jul 28 13:02:44.054839 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.054819    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"]
Jul 28 13:02:44.056510 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.056493    2569 scope.go:117] "RemoveContainer" containerID="082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd"
Jul 28 13:02:44.056772 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:02:44.056754    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd\": container with ID starting with 082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd not found: ID does not exist" containerID="082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd"
Jul 28 13:02:44.056811 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.056782    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd"} err="failed to get container status \"082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd\": rpc error: code = NotFound desc = could not find container \"082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd\": container with ID starting with 082c088e727e1265a08194f7260928143fc50a8571d4abd7db0fcd3eeb9429bd not found: ID does not exist"
Jul 28 13:02:44.056811 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.056800    2569 scope.go:117] "RemoveContainer" containerID="53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a"
Jul 28 13:02:44.057047 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:02:44.057031    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a\": container with ID starting with 53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a not found: ID does not exist" containerID="53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a"
Jul 28 13:02:44.057089 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.057053    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a"} err="failed to get container status \"53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a\": rpc error: code = NotFound desc = could not find container \"53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a\": container with ID starting with 53aa6660cb4d7b0b287b59fc8b9dd0b091df1a6f39f60524a8fb1b1e7fe7960a not found: ID does not exist"
Jul 28 13:02:44.057089 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.057068    2569 scope.go:117] "RemoveContainer" containerID="2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042"
Jul 28 13:02:44.057315 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:02:44.057298    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042\": container with ID starting with 2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042 not found: ID does not exist" containerID="2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042"
Jul 28 13:02:44.057351 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.057321    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042"} err="failed to get container status \"2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042\": rpc error: code = NotFound desc = could not find container \"2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042\": container with ID starting with 2eeb892c01a3e88bd06a6b6f1decae67eb19ee00e5fb63e8bb7df8fa8b061042 not found: ID does not exist"
Jul 28 13:02:44.060328 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.060310    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-675cc7cb4-4qvhs"]
Jul 28 13:02:44.253821 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:02:44.253788    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" path="/var/lib/kubelet/pods/2cdf12ee-7c10-4638-bece-8fa0305dec07/volumes"
Jul 28 13:04:09.385385 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:09.385349    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"]
Jul 28 13:04:09.385899 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:09.385669    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" containerID="cri-o://f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6" gracePeriod=30
Jul 28 13:04:09.385899 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:09.385692    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kube-rbac-proxy" containerID="cri-o://af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df" gracePeriod=30
Jul 28 13:04:10.428420 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:10.428385    2569 generic.go:356] "Generic (PLEG): container finished" podID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerID="af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df" exitCode=2
Jul 28 13:04:10.428799 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:10.428459    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" event={"ID":"89efd3d2-ceb4-4200-8b11-24b899ce188a","Type":"ContainerDied","Data":"af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df"}
Jul 28 13:04:10.612473 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:10.612431    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 13:04:10.616801 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:10.616773    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:04:14.133335 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.133309    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:04:14.203955 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.203919    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89efd3d2-ceb4-4200-8b11-24b899ce188a-proxy-tls\") pod \"89efd3d2-ceb4-4200-8b11-24b899ce188a\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") "
Jul 28 13:04:14.203955 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.203961    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzvnn\" (UniqueName: \"kubernetes.io/projected/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-api-access-wzvnn\") pod \"89efd3d2-ceb4-4200-8b11-24b899ce188a\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") "
Jul 28 13:04:14.204242 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.204020    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89efd3d2-ceb4-4200-8b11-24b899ce188a-kserve-provision-location\") pod \"89efd3d2-ceb4-4200-8b11-24b899ce188a\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") "
Jul 28 13:04:14.204242 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.204046    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-cabundle-cert\") pod \"89efd3d2-ceb4-4200-8b11-24b899ce188a\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") "
Jul 28 13:04:14.204242 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.204109    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-rbac-proxy-sar-config\") pod \"89efd3d2-ceb4-4200-8b11-24b899ce188a\" (UID: \"89efd3d2-ceb4-4200-8b11-24b899ce188a\") "
Jul 28 13:04:14.204409 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.204348    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89efd3d2-ceb4-4200-8b11-24b899ce188a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89efd3d2-ceb4-4200-8b11-24b899ce188a" (UID: "89efd3d2-ceb4-4200-8b11-24b899ce188a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:04:14.204499 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.204470    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89efd3d2-ceb4-4200-8b11-24b899ce188a" (UID: "89efd3d2-ceb4-4200-8b11-24b899ce188a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:04:14.204566 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.204493    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89efd3d2-ceb4-4200-8b11-24b899ce188a" (UID: "89efd3d2-ceb4-4200-8b11-24b899ce188a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:04:14.206132 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.206106    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89efd3d2-ceb4-4200-8b11-24b899ce188a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89efd3d2-ceb4-4200-8b11-24b899ce188a" (UID: "89efd3d2-ceb4-4200-8b11-24b899ce188a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:04:14.206259 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.206237    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-api-access-wzvnn" (OuterVolumeSpecName: "kube-api-access-wzvnn") pod "89efd3d2-ceb4-4200-8b11-24b899ce188a" (UID: "89efd3d2-ceb4-4200-8b11-24b899ce188a"). InnerVolumeSpecName "kube-api-access-wzvnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:04:14.305459 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.305427    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89efd3d2-ceb4-4200-8b11-24b899ce188a-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:04:14.305459 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.305456    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzvnn\" (UniqueName: \"kubernetes.io/projected/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-api-access-wzvnn\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:04:14.305459 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.305467    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89efd3d2-ceb4-4200-8b11-24b899ce188a-kserve-provision-location\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:04:14.305682 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.305476    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-cabundle-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:04:14.305682 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.305485    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89efd3d2-ceb4-4200-8b11-24b899ce188a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:04:14.451349 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.451227    2569 generic.go:356] "Generic (PLEG): container finished" podID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerID="f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6" exitCode=0
Jul 28 13:04:14.451349 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.451349    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" event={"ID":"89efd3d2-ceb4-4200-8b11-24b899ce188a","Type":"ContainerDied","Data":"f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6"}
Jul 28 13:04:14.451598 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.451528    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"
Jul 28 13:04:14.451659 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.451595    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk" event={"ID":"89efd3d2-ceb4-4200-8b11-24b899ce188a","Type":"ContainerDied","Data":"27e4c8a2c02aee13d158620f6dcecd1b93b7a886583124ec067d4124b5032044"}
Jul 28 13:04:14.451659 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.451631    2569 scope.go:117] "RemoveContainer" containerID="af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df"
Jul 28 13:04:14.460899 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.460879    2569 scope.go:117] "RemoveContainer" containerID="f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6"
Jul 28 13:04:14.469474 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.469192    2569 scope.go:117] "RemoveContainer" containerID="e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b"
Jul 28 13:04:14.471468 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.471445    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"]
Jul 28 13:04:14.477330 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.477270    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-c6xvk"]
Jul 28 13:04:14.477412 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.477350    2569 scope.go:117] "RemoveContainer" containerID="af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df"
Jul 28 13:04:14.477596 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:04:14.477579    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df\": container with ID starting with af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df not found: ID does not exist" containerID="af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df"
Jul 28 13:04:14.477669 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.477608    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df"} err="failed to get container status \"af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df\": rpc error: code = NotFound desc = could not find container \"af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df\": container with ID starting with af2ddd091d53178f2f658ccdd18590584a389c50eb8b25e5413fabcf0540c1df not found: ID does not exist"
Jul 28 13:04:14.477669 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.477632    2569 scope.go:117] "RemoveContainer" containerID="f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6"
Jul 28 13:04:14.477894 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:04:14.477874    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6\": container with ID starting with f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6 not found: ID does not exist" containerID="f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6"
Jul 28 13:04:14.477946 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.477899    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6"} err="failed to get container status \"f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6\": rpc error: code = NotFound desc = could not find container \"f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6\": container with ID starting with f1720d7cff0b7b88d993ef85e66f380eb89b56196ab6155bc405292a3a99fac6 not found: ID does not exist"
Jul 28 13:04:14.477946 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.477915    2569 scope.go:117] "RemoveContainer" containerID="e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b"
Jul 28 13:04:14.478206 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:04:14.478190    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b\": container with ID starting with e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b not found: ID does not exist" containerID="e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b"
Jul 28 13:04:14.478260 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.478210    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b"} err="failed to get container status \"e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b\": rpc error: code = NotFound desc = could not find container \"e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b\": container with ID starting with e4bdf66dd0e216f3cf35c71b5305dfde0dc022fe7658f96bdfb38eaa8acc3e6b not found: ID does not exist"
Jul 28 13:04:14.560515 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560423    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"]
Jul 28 13:04:14.560792 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560780    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container"
Jul 28 13:04:14.560835 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560794    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container"
Jul 28 13:04:14.560835 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560812    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="storage-initializer"
Jul 28 13:04:14.560835 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560818    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="storage-initializer"
Jul 28 13:04:14.560835 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560826    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container"
Jul 28 13:04:14.560835 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560831    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container"
Jul 28 13:04:14.561015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560841    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="storage-initializer"
Jul 28 13:04:14.561015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560847    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="storage-initializer"
Jul 28 13:04:14.561015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560852    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kube-rbac-proxy"
Jul 28 13:04:14.561015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560857    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kube-rbac-proxy"
Jul 28 13:04:14.561015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560864    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kube-rbac-proxy"
Jul 28 13:04:14.561015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560869    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kube-rbac-proxy"
Jul 28 13:04:14.561015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560920    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kserve-container"
Jul 28 13:04:14.561015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560929    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" containerName="kube-rbac-proxy"
Jul 28 13:04:14.561015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560936    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kserve-container"
Jul 28 13:04:14.561015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.560943    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cdf12ee-7c10-4638-bece-8fa0305dec07" containerName="kube-rbac-proxy"
Jul 28 13:04:14.564179 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.564158    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.567082 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.567049    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:04:14.567082 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.567086    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rtw6d\""
Jul 28 13:04:14.567453 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.567435    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 13:04:14.567596 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.567580    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:04:14.567802 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.567781    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:04:14.567907 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.567790    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:04:14.567970 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.567958    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 13:04:14.574846 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.574821    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"]
Jul 28 13:04:14.607517 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.607486    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.607694 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.607538    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.607694 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.607586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-cabundle-cert\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.607694 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.607622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9rqp\" (UniqueName: \"kubernetes.io/projected/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-api-access-n9rqp\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.607694 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.607658    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-proxy-tls\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.708839 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.708802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.709027 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.708852    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.709027 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.708877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-cabundle-cert\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.709027 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.708907    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9rqp\" (UniqueName: \"kubernetes.io/projected/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-api-access-n9rqp\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.709027 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.708926    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-proxy-tls\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.709274 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:04:14.709092    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 28 13:04:14.709274 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:04:14.709191    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-proxy-tls podName:3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6 nodeName:}" failed. No retries permitted until 2026-07-28 13:04:15.209170457 +0000 UTC m=+1823.424687104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-proxy-tls") pod "isvc-canary-stop-predictor-59777d5f7c-gg6g9" (UID: "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 28 13:04:14.709459 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.709440    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.709668 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.709646    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-cabundle-cert\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.709733 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.709691    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:14.717236 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:14.717212    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9rqp\" (UniqueName: \"kubernetes.io/projected/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-api-access-n9rqp\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:15.213463 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:15.213425    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-proxy-tls\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:15.215973 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:15.215941    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-proxy-tls\") pod \"isvc-canary-stop-predictor-59777d5f7c-gg6g9\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:15.476015 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:15.475911    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:15.609038 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:15.609008    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"]
Jul 28 13:04:15.611011 ip-10-0-142-139 kubenswrapper[2569]: W0728 13:04:15.610950    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a7040_f1bf_4b0b_b9e0_c631469cfbb6.slice/crio-445e6875b75ebbc3f37e7d9029f14678b43bf16ff9e38eca77a78cd33e47b4d5 WatchSource:0}: Error finding container 445e6875b75ebbc3f37e7d9029f14678b43bf16ff9e38eca77a78cd33e47b4d5: Status 404 returned error can't find the container with id 445e6875b75ebbc3f37e7d9029f14678b43bf16ff9e38eca77a78cd33e47b4d5
Jul 28 13:04:16.253163 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:16.253128    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89efd3d2-ceb4-4200-8b11-24b899ce188a" path="/var/lib/kubelet/pods/89efd3d2-ceb4-4200-8b11-24b899ce188a/volumes"
Jul 28 13:04:16.463182 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:16.463148    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" event={"ID":"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6","Type":"ContainerStarted","Data":"21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090"}
Jul 28 13:04:16.463363 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:16.463202    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" event={"ID":"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6","Type":"ContainerStarted","Data":"445e6875b75ebbc3f37e7d9029f14678b43bf16ff9e38eca77a78cd33e47b4d5"}
Jul 28 13:04:19.478928 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:19.478894    2569 generic.go:356] "Generic (PLEG): container finished" podID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerID="21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090" exitCode=0
Jul 28 13:04:19.479333 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:19.478969    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" event={"ID":"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6","Type":"ContainerDied","Data":"21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090"}
Jul 28 13:04:20.486150 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:20.486115    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" event={"ID":"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6","Type":"ContainerStarted","Data":"8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9"}
Jul 28 13:04:20.486552 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:20.486180    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" event={"ID":"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6","Type":"ContainerStarted","Data":"eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27"}
Jul 28 13:04:20.486552 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:20.486451    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:20.486628 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:20.486587    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:20.487982 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:20.487955    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:04:20.505255 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:20.505182    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podStartSLOduration=2.79795336 podStartE2EDuration="6.50515341s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.70720005s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:04:20.504001633 +0000 UTC m=+1828.719518279" watchObservedRunningTime="2026-07-28 13:04:20.50515341 +0000 UTC m=+1828.720670063"
Jul 28 13:04:21.491129 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:21.491088    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:04:26.496604 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:26.496574    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:04:26.497112 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:26.497083    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:04:36.497863 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:36.497822    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:04:46.497087 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:46.497042    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:04:56.498054 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:04:56.498011    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:05:06.497290 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:06.497248    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:05:16.497786 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:16.497700    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:05:26.497341 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:26.497297    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:05:36.497970 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:36.497941    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:05:44.573957 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:44.573908    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"]
Jul 28 13:05:44.574453 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:44.574216    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" containerID="cri-o://eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27" gracePeriod=30
Jul 28 13:05:44.574614 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:44.574572    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kube-rbac-proxy" containerID="cri-o://8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9" gracePeriod=30
Jul 28 13:05:44.863405 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:44.863319    2569 generic.go:356] "Generic (PLEG): container finished" podID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerID="8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9" exitCode=2
Jul 28 13:05:44.863553 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:44.863395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" event={"ID":"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6","Type":"ContainerDied","Data":"8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9"}
Jul 28 13:05:46.492108 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:46.492060    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 13:05:46.497470 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:46.497444    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:05:49.124552 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.124524    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:05:49.255050 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.254961    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-cabundle-cert\") pod \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") "
Jul 28 13:05:49.255050 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.255006    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9rqp\" (UniqueName: \"kubernetes.io/projected/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-api-access-n9rqp\") pod \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") "
Jul 28 13:05:49.255050 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.255044    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-rbac-proxy-sar-config\") pod \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") "
Jul 28 13:05:49.255331 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.255069    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kserve-provision-location\") pod \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") "
Jul 28 13:05:49.255331 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.255147    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-proxy-tls\") pod \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\" (UID: \"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6\") "
Jul 28 13:05:49.255494 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.255295    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" (UID: "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:05:49.255494 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.255427    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" (UID: "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:05:49.255494 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.255403    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" (UID: "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:05:49.257186 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.257160    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" (UID: "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:05:49.257310 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.257208    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-api-access-n9rqp" (OuterVolumeSpecName: "kube-api-access-n9rqp") pod "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" (UID: "3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6"). InnerVolumeSpecName "kube-api-access-n9rqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:05:49.355739 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.355705    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:05:49.355739 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.355732    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-cabundle-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:05:49.355739 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.355743    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9rqp\" (UniqueName: \"kubernetes.io/projected/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-api-access-n9rqp\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:05:49.355962 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.355752    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:05:49.355962 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.355761    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6-kserve-provision-location\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:05:49.891670 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.891631    2569 generic.go:356] "Generic (PLEG): container finished" podID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerID="eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27" exitCode=0
Jul 28 13:05:49.891881 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.891694    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" event={"ID":"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6","Type":"ContainerDied","Data":"eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27"}
Jul 28 13:05:49.891881 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.891708    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"
Jul 28 13:05:49.891881 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.891762    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9" event={"ID":"3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6","Type":"ContainerDied","Data":"445e6875b75ebbc3f37e7d9029f14678b43bf16ff9e38eca77a78cd33e47b4d5"}
Jul 28 13:05:49.891881 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.891790    2569 scope.go:117] "RemoveContainer" containerID="8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9"
Jul 28 13:05:49.900736 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.900719    2569 scope.go:117] "RemoveContainer" containerID="eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27"
Jul 28 13:05:49.907961 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.907944    2569 scope.go:117] "RemoveContainer" containerID="21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090"
Jul 28 13:05:49.915166 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.915144    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"]
Jul 28 13:05:49.915725 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.915711    2569 scope.go:117] "RemoveContainer" containerID="8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9"
Jul 28 13:05:49.915967 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:05:49.915947    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9\": container with ID starting with 8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9 not found: ID does not exist" containerID="8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9"
Jul 28 13:05:49.916072 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.915974    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9"} err="failed to get container status \"8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9\": rpc error: code = NotFound desc = could not find container \"8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9\": container with ID starting with 8741042dd0bd515d6046be17c358116b893d3b8f558fc329b46fa9e9a8c539a9 not found: ID does not exist"
Jul 28 13:05:49.916072 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.916004    2569 scope.go:117] "RemoveContainer" containerID="eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27"
Jul 28 13:05:49.916254 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:05:49.916238    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27\": container with ID starting with eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27 not found: ID does not exist" containerID="eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27"
Jul 28 13:05:49.916298 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.916259    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27"} err="failed to get container status \"eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27\": rpc error: code = NotFound desc = could not find container \"eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27\": container with ID starting with eeac4d014d5b34f43dae317e53aae15dbe8b7a963d1ff9f08a6e0ef63a862f27 not found: ID does not exist"
Jul 28 13:05:49.916298 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.916273    2569 scope.go:117] "RemoveContainer" containerID="21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090"
Jul 28 13:05:49.916496 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:05:49.916478    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090\": container with ID starting with 21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090 not found: ID does not exist" containerID="21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090"
Jul 28 13:05:49.916539 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.916521    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090"} err="failed to get container status \"21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090\": rpc error: code = NotFound desc = could not find container \"21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090\": container with ID starting with 21890ee09fd97ed820efdb6a4de93dc7319548694d76c44c2e9f5624ab3cf090 not found: ID does not exist"
Jul 28 13:05:49.919122 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:49.919103    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59777d5f7c-gg6g9"]
Jul 28 13:05:50.252187 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:50.252157    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" path="/var/lib/kubelet/pods/3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6/volumes"
Jul 28 13:05:54.867231 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.867198    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"]
Jul 28 13:05:54.868347 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.868316    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="storage-initializer"
Jul 28 13:05:54.868347 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.868348    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="storage-initializer"
Jul 28 13:05:54.868544 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.868390    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container"
Jul 28 13:05:54.868544 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.868401    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container"
Jul 28 13:05:54.868544 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.868461    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kube-rbac-proxy"
Jul 28 13:05:54.868544 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.868472    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kube-rbac-proxy"
Jul 28 13:05:54.868784 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.868766    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kube-rbac-proxy"
Jul 28 13:05:54.868848 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.868794    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7a7040-f1bf-4b0b-b9e0-c631469cfbb6" containerName="kserve-container"
Jul 28 13:05:54.875112 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.875086    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:54.877756 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.877728    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d919a2-kube-rbac-proxy-sar-config\""
Jul 28 13:05:54.877874 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.877810    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:05:54.877874 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.877733    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d919a2-predictor-serving-cert\""
Jul 28 13:05:54.878012 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.877898    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rtw6d\""
Jul 28 13:05:54.878476 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.878451    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:05:54.878766 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.878742    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:05:54.878869 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.878790    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:05:54.882609 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:54.882507    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"]
Jul 28 13:05:55.002806 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.002772    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79d744b3-265a-4759-bc1d-2ddfc47275b8-proxy-tls\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.002999 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.002811    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.002999 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.002851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79d744b3-265a-4759-bc1d-2ddfc47275b8-kserve-provision-location\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.002999 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.002868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjs8\" (UniqueName: \"kubernetes.io/projected/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-api-access-jsjs8\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.002999 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.002896    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-cabundle-cert\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.103354 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.103318    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79d744b3-265a-4759-bc1d-2ddfc47275b8-kserve-provision-location\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.103354 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.103356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjs8\" (UniqueName: \"kubernetes.io/projected/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-api-access-jsjs8\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.103552 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.103405    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-cabundle-cert\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.103552 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.103476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79d744b3-265a-4759-bc1d-2ddfc47275b8-proxy-tls\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.103552 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.103511    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.105344 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.105319    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79d744b3-265a-4759-bc1d-2ddfc47275b8-kserve-provision-location\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.105586 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.105564    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-cabundle-cert\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.106527 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.106505    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.107293 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.107276    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79d744b3-265a-4759-bc1d-2ddfc47275b8-proxy-tls\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.111361 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.111342    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjs8\" (UniqueName: \"kubernetes.io/projected/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-api-access-jsjs8\") pod \"isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") " pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.186172 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.186092    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:05:55.317467 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.317438    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"]
Jul 28 13:05:55.319083 ip-10-0-142-139 kubenswrapper[2569]: W0728 13:05:55.319056    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d744b3_265a_4759_bc1d_2ddfc47275b8.slice/crio-6e1e1a4d7fa33e8593c166d92ea9cbe02ff8507adf6206b31d60a5b7f318e472 WatchSource:0}: Error finding container 6e1e1a4d7fa33e8593c166d92ea9cbe02ff8507adf6206b31d60a5b7f318e472: Status 404 returned error can't find the container with id 6e1e1a4d7fa33e8593c166d92ea9cbe02ff8507adf6206b31d60a5b7f318e472
Jul 28 13:05:55.920744 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.920713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" event={"ID":"79d744b3-265a-4759-bc1d-2ddfc47275b8","Type":"ContainerStarted","Data":"8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b"}
Jul 28 13:05:55.921146 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:55.920768    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" event={"ID":"79d744b3-265a-4759-bc1d-2ddfc47275b8","Type":"ContainerStarted","Data":"6e1e1a4d7fa33e8593c166d92ea9cbe02ff8507adf6206b31d60a5b7f318e472"}
Jul 28 13:05:59.940544 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:59.940507    2569 generic.go:356] "Generic (PLEG): container finished" podID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerID="8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b" exitCode=0
Jul 28 13:05:59.941108 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:05:59.940582    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" event={"ID":"79d744b3-265a-4759-bc1d-2ddfc47275b8","Type":"ContainerDied","Data":"8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b"}
Jul 28 13:06:00.947535 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:00.947501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" event={"ID":"79d744b3-265a-4759-bc1d-2ddfc47275b8","Type":"ContainerStarted","Data":"2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509"}
Jul 28 13:06:00.948045 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:00.947569    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" event={"ID":"79d744b3-265a-4759-bc1d-2ddfc47275b8","Type":"ContainerStarted","Data":"91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442"}
Jul 28 13:06:00.948045 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:00.947790    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:06:00.968192 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:00.968144    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podStartSLOduration=3.27108957 podStartE2EDuration="6.968129087s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.697039517s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:06:00.966689151 +0000 UTC m=+1929.182205833" watchObservedRunningTime="2026-07-28 13:06:00.968129087 +0000 UTC m=+1929.183645738"
Jul 28 13:06:01.952326 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:01.952291    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:06:01.953784 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:01.953760    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:06:02.957186 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:02.957142    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:06:07.962192 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:07.962159    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:06:07.962795 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:07.962768    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:06:17.963056 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:17.963018    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:06:27.963048 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:27.963007    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:06:37.962736 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:37.962698    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:06:47.962824 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:47.962777    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:06:57.963295 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:06:57.963251    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:07:07.963699 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:07:07.963648    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:07:12.249154 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:07:12.249114    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:07:22.253394 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:07:22.253364    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:07:56.112852 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:07:56.112813    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"]
Jul 28 13:07:56.113691 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:07:56.113089    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container" containerID="cri-o://91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442" gracePeriod=30
Jul 28 13:07:56.113691 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:07:56.113151    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kube-rbac-proxy" containerID="cri-o://2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509" gracePeriod=30
Jul 28 13:07:56.486121 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:07:56.486091    2569 generic.go:356] "Generic (PLEG): container finished" podID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerID="2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509" exitCode=2
Jul 28 13:07:56.486283 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:07:56.486135    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" event={"ID":"79d744b3-265a-4759-bc1d-2ddfc47275b8","Type":"ContainerDied","Data":"2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509"}
Jul 28 13:07:57.957322 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:07:57.957282    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 28 13:08:00.658682 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.658658    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:08:00.737497 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.737415    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-rbac-proxy-sar-config\") pod \"79d744b3-265a-4759-bc1d-2ddfc47275b8\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") "
Jul 28 13:08:00.737497 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.737452    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79d744b3-265a-4759-bc1d-2ddfc47275b8-proxy-tls\") pod \"79d744b3-265a-4759-bc1d-2ddfc47275b8\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") "
Jul 28 13:08:00.737497 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.737481    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsjs8\" (UniqueName: \"kubernetes.io/projected/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-api-access-jsjs8\") pod \"79d744b3-265a-4759-bc1d-2ddfc47275b8\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") "
Jul 28 13:08:00.737775 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.737536    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-cabundle-cert\") pod \"79d744b3-265a-4759-bc1d-2ddfc47275b8\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") "
Jul 28 13:08:00.737775 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.737575    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79d744b3-265a-4759-bc1d-2ddfc47275b8-kserve-provision-location\") pod \"79d744b3-265a-4759-bc1d-2ddfc47275b8\" (UID: \"79d744b3-265a-4759-bc1d-2ddfc47275b8\") "
Jul 28 13:08:00.737930 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.737903    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79d744b3-265a-4759-bc1d-2ddfc47275b8" (UID: "79d744b3-265a-4759-bc1d-2ddfc47275b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:00.737979 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.737906    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d744b3-265a-4759-bc1d-2ddfc47275b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79d744b3-265a-4759-bc1d-2ddfc47275b8" (UID: "79d744b3-265a-4759-bc1d-2ddfc47275b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:08:00.738045 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.738018    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79d744b3-265a-4759-bc1d-2ddfc47275b8" (UID: "79d744b3-265a-4759-bc1d-2ddfc47275b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:00.739497 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.739474    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-api-access-jsjs8" (OuterVolumeSpecName: "kube-api-access-jsjs8") pod "79d744b3-265a-4759-bc1d-2ddfc47275b8" (UID: "79d744b3-265a-4759-bc1d-2ddfc47275b8"). InnerVolumeSpecName "kube-api-access-jsjs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:08:00.739571 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.739504    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d744b3-265a-4759-bc1d-2ddfc47275b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79d744b3-265a-4759-bc1d-2ddfc47275b8" (UID: "79d744b3-265a-4759-bc1d-2ddfc47275b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:08:00.839059 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.839028    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-cabundle-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:08:00.839059 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.839054    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79d744b3-265a-4759-bc1d-2ddfc47275b8-kserve-provision-location\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:08:00.839059 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.839064    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:08:00.839258 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.839076    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79d744b3-265a-4759-bc1d-2ddfc47275b8-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:08:00.839258 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:00.839084    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsjs8\" (UniqueName: \"kubernetes.io/projected/79d744b3-265a-4759-bc1d-2ddfc47275b8-kube-api-access-jsjs8\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:08:01.383860 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.383827    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"]
Jul 28 13:08:01.384216 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.384203    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container"
Jul 28 13:08:01.384268 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.384218    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container"
Jul 28 13:08:01.384268 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.384225    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kube-rbac-proxy"
Jul 28 13:08:01.384268 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.384231    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kube-rbac-proxy"
Jul 28 13:08:01.384268 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.384244    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="storage-initializer"
Jul 28 13:08:01.384268 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.384251    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="storage-initializer"
Jul 28 13:08:01.384425 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.384315    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kserve-container"
Jul 28 13:08:01.384425 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.384326    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerName="kube-rbac-proxy"
Jul 28 13:08:01.387603 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.387587    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.390123 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.390101    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-4e2983-dockercfg-vfrzx\""
Jul 28 13:08:01.390275 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.390210    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4e2983-predictor-serving-cert\""
Jul 28 13:08:01.390275 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.390228    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-4e2983\""
Jul 28 13:08:01.390617 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.390598    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4e2983-kube-rbac-proxy-sar-config\""
Jul 28 13:08:01.398230 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.398209    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"]
Jul 28 13:08:01.444908 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.444882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.445119 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.444927    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49f8efb7-37ed-4442-8959-e7b41071ae27-proxy-tls\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.445119 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.444956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-cabundle-cert\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.445119 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.445066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k7lx\" (UniqueName: \"kubernetes.io/projected/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-api-access-6k7lx\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.445119 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.445099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49f8efb7-37ed-4442-8959-e7b41071ae27-kserve-provision-location\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.510184 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.510149    2569 generic.go:356] "Generic (PLEG): container finished" podID="79d744b3-265a-4759-bc1d-2ddfc47275b8" containerID="91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442" exitCode=0
Jul 28 13:08:01.510326 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.510206    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"
Jul 28 13:08:01.510326 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.510230    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" event={"ID":"79d744b3-265a-4759-bc1d-2ddfc47275b8","Type":"ContainerDied","Data":"91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442"}
Jul 28 13:08:01.510326 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.510286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w" event={"ID":"79d744b3-265a-4759-bc1d-2ddfc47275b8","Type":"ContainerDied","Data":"6e1e1a4d7fa33e8593c166d92ea9cbe02ff8507adf6206b31d60a5b7f318e472"}
Jul 28 13:08:01.510326 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.510313    2569 scope.go:117] "RemoveContainer" containerID="2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509"
Jul 28 13:08:01.520741 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.520723    2569 scope.go:117] "RemoveContainer" containerID="91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442"
Jul 28 13:08:01.528549 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.528529    2569 scope.go:117] "RemoveContainer" containerID="8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b"
Jul 28 13:08:01.535670 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.535542    2569 scope.go:117] "RemoveContainer" containerID="2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509"
Jul 28 13:08:01.535818 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:08:01.535790    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509\": container with ID starting with 2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509 not found: ID does not exist" containerID="2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509"
Jul 28 13:08:01.535867 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.535827    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509"} err="failed to get container status \"2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509\": rpc error: code = NotFound desc = could not find container \"2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509\": container with ID starting with 2112e99914c21881aa7b453a5c5d5787fb4621a34f1af50faa52397d05cfd509 not found: ID does not exist"
Jul 28 13:08:01.535867 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.535844    2569 scope.go:117] "RemoveContainer" containerID="91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442"
Jul 28 13:08:01.536115 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:08:01.536097    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442\": container with ID starting with 91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442 not found: ID does not exist" containerID="91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442"
Jul 28 13:08:01.536172 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.536120    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442"} err="failed to get container status \"91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442\": rpc error: code = NotFound desc = could not find container \"91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442\": container with ID starting with 91c4b0eeafeed88e2eaf5f31a4baa5e983cce363062b61c87cbb03255b8e4442 not found: ID does not exist"
Jul 28 13:08:01.536172 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.536135    2569 scope.go:117] "RemoveContainer" containerID="8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b"
Jul 28 13:08:01.536347 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:08:01.536334    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b\": container with ID starting with 8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b not found: ID does not exist" containerID="8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b"
Jul 28 13:08:01.536396 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.536352    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b"} err="failed to get container status \"8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b\": rpc error: code = NotFound desc = could not find container \"8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b\": container with ID starting with 8a758da42976f23e5205e20d6c2d33157bde5bc0fca9e5a68f202a6bb6b5c72b not found: ID does not exist"
Jul 28 13:08:01.545923 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.545893    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"]
Jul 28 13:08:01.546211 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.546193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-cabundle-cert\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.546273 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.546237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k7lx\" (UniqueName: \"kubernetes.io/projected/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-api-access-6k7lx\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.546273 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.546261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49f8efb7-37ed-4442-8959-e7b41071ae27-kserve-provision-location\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.546463 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.546435    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.546576 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.546506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49f8efb7-37ed-4442-8959-e7b41071ae27-proxy-tls\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.546576 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.546558    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49f8efb7-37ed-4442-8959-e7b41071ae27-kserve-provision-location\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.546670 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:08:01.546629    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-serving-cert: secret "isvc-init-fail-4e2983-predictor-serving-cert" not found
Jul 28 13:08:01.546706 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:08:01.546680    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f8efb7-37ed-4442-8959-e7b41071ae27-proxy-tls podName:49f8efb7-37ed-4442-8959-e7b41071ae27 nodeName:}" failed. No retries permitted until 2026-07-28 13:08:02.046667319 +0000 UTC m=+2050.262183949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/49f8efb7-37ed-4442-8959-e7b41071ae27-proxy-tls") pod "isvc-init-fail-4e2983-predictor-79ccd5778-ntk66" (UID: "49f8efb7-37ed-4442-8959-e7b41071ae27") : secret "isvc-init-fail-4e2983-predictor-serving-cert" not found
Jul 28 13:08:01.546941 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.546921    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-cabundle-cert\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.547042 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.547028    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:01.563636 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.563613    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d919a2-predictor-7fd74c9d69-bnk8w"]
Jul 28 13:08:01.567087 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:01.567064    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k7lx\" (UniqueName: \"kubernetes.io/projected/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-api-access-6k7lx\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:02.051586 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:02.051542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49f8efb7-37ed-4442-8959-e7b41071ae27-proxy-tls\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:02.054211 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:02.054176    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49f8efb7-37ed-4442-8959-e7b41071ae27-proxy-tls\") pod \"isvc-init-fail-4e2983-predictor-79ccd5778-ntk66\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") " pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:02.251803 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:02.251766    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79d744b3-265a-4759-bc1d-2ddfc47275b8" path="/var/lib/kubelet/pods/79d744b3-265a-4759-bc1d-2ddfc47275b8/volumes"
Jul 28 13:08:02.298834 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:02.298801    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:02.429353 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:02.429322    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"]
Jul 28 13:08:02.432733 ip-10-0-142-139 kubenswrapper[2569]: W0728 13:08:02.432709    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f8efb7_37ed_4442_8959_e7b41071ae27.slice/crio-b00a0942499a6552ed50948851ede4634457034aa3010cb770c42f2473bd194e WatchSource:0}: Error finding container b00a0942499a6552ed50948851ede4634457034aa3010cb770c42f2473bd194e: Status 404 returned error can't find the container with id b00a0942499a6552ed50948851ede4634457034aa3010cb770c42f2473bd194e
Jul 28 13:08:02.516393 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:02.516364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66" event={"ID":"49f8efb7-37ed-4442-8959-e7b41071ae27","Type":"ContainerStarted","Data":"b6298272e88e21f21c2f99f0bda975edd96c00a2058798fd8cb0f470e719e88e"}
Jul 28 13:08:02.516556 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:02.516429    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66" event={"ID":"49f8efb7-37ed-4442-8959-e7b41071ae27","Type":"ContainerStarted","Data":"b00a0942499a6552ed50948851ede4634457034aa3010cb770c42f2473bd194e"}
Jul 28 13:08:06.542468 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:06.542388    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4e2983-predictor-79ccd5778-ntk66_49f8efb7-37ed-4442-8959-e7b41071ae27/storage-initializer/0.log"
Jul 28 13:08:06.542468 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:06.542429    2569 generic.go:356] "Generic (PLEG): container finished" podID="49f8efb7-37ed-4442-8959-e7b41071ae27" containerID="b6298272e88e21f21c2f99f0bda975edd96c00a2058798fd8cb0f470e719e88e" exitCode=1
Jul 28 13:08:06.542874 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:06.542490    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66" event={"ID":"49f8efb7-37ed-4442-8959-e7b41071ae27","Type":"ContainerDied","Data":"b6298272e88e21f21c2f99f0bda975edd96c00a2058798fd8cb0f470e719e88e"}
Jul 28 13:08:07.549374 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:07.549342    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4e2983-predictor-79ccd5778-ntk66_49f8efb7-37ed-4442-8959-e7b41071ae27/storage-initializer/0.log"
Jul 28 13:08:07.549824 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:07.549456    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66" event={"ID":"49f8efb7-37ed-4442-8959-e7b41071ae27","Type":"ContainerStarted","Data":"fa57a2de6d21a8e3ea0222be57f2c7442c422271c264f1e4b22739844eb80934"}
Jul 28 13:08:10.563774 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:10.563745    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4e2983-predictor-79ccd5778-ntk66_49f8efb7-37ed-4442-8959-e7b41071ae27/storage-initializer/1.log"
Jul 28 13:08:10.564187 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:10.564135    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4e2983-predictor-79ccd5778-ntk66_49f8efb7-37ed-4442-8959-e7b41071ae27/storage-initializer/0.log"
Jul 28 13:08:10.564187 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:10.564166    2569 generic.go:356] "Generic (PLEG): container finished" podID="49f8efb7-37ed-4442-8959-e7b41071ae27" containerID="fa57a2de6d21a8e3ea0222be57f2c7442c422271c264f1e4b22739844eb80934" exitCode=1
Jul 28 13:08:10.564265 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:10.564193    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66" event={"ID":"49f8efb7-37ed-4442-8959-e7b41071ae27","Type":"ContainerDied","Data":"fa57a2de6d21a8e3ea0222be57f2c7442c422271c264f1e4b22739844eb80934"}
Jul 28 13:08:10.564265 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:10.564246    2569 scope.go:117] "RemoveContainer" containerID="b6298272e88e21f21c2f99f0bda975edd96c00a2058798fd8cb0f470e719e88e"
Jul 28 13:08:10.564649 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:10.564632    2569 scope.go:117] "RemoveContainer" containerID="b6298272e88e21f21c2f99f0bda975edd96c00a2058798fd8cb0f470e719e88e"
Jul 28 13:08:10.574953 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:08:10.574932    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-4e2983-predictor-79ccd5778-ntk66_kserve-ci-e2e-test(49f8efb7-37ed-4442-8959-e7b41071ae27)\"" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66" podUID="49f8efb7-37ed-4442-8959-e7b41071ae27"
Jul 28 13:08:10.575199 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:08:10.575175    2569 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-4e2983-predictor-79ccd5778-ntk66_kserve-ci-e2e-test_49f8efb7-37ed-4442-8959-e7b41071ae27_0 in pod sandbox b00a0942499a6552ed50948851ede4634457034aa3010cb770c42f2473bd194e from index: no such id: 'b6298272e88e21f21c2f99f0bda975edd96c00a2058798fd8cb0f470e719e88e'" containerID="b6298272e88e21f21c2f99f0bda975edd96c00a2058798fd8cb0f470e719e88e"
Jul 28 13:08:10.575249 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:10.575206    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6298272e88e21f21c2f99f0bda975edd96c00a2058798fd8cb0f470e719e88e"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-4e2983-predictor-79ccd5778-ntk66_kserve-ci-e2e-test_49f8efb7-37ed-4442-8959-e7b41071ae27_0 in pod sandbox b00a0942499a6552ed50948851ede4634457034aa3010cb770c42f2473bd194e from index: no such id: 'b6298272e88e21f21c2f99f0bda975edd96c00a2058798fd8cb0f470e719e88e'"
Jul 28 13:08:11.405919 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.405882    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"]
Jul 28 13:08:11.569783 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.569748    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4e2983-predictor-79ccd5778-ntk66_49f8efb7-37ed-4442-8959-e7b41071ae27/storage-initializer/1.log"
Jul 28 13:08:11.704874 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.704853    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4e2983-predictor-79ccd5778-ntk66_49f8efb7-37ed-4442-8959-e7b41071ae27/storage-initializer/1.log"
Jul 28 13:08:11.708365 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.708348    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:11.842238 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.842206    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-rbac-proxy-sar-config\") pod \"49f8efb7-37ed-4442-8959-e7b41071ae27\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") "
Jul 28 13:08:11.842238 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.842239    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49f8efb7-37ed-4442-8959-e7b41071ae27-kserve-provision-location\") pod \"49f8efb7-37ed-4442-8959-e7b41071ae27\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") "
Jul 28 13:08:11.842510 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.842284    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49f8efb7-37ed-4442-8959-e7b41071ae27-proxy-tls\") pod \"49f8efb7-37ed-4442-8959-e7b41071ae27\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") "
Jul 28 13:08:11.842510 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.842315    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k7lx\" (UniqueName: \"kubernetes.io/projected/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-api-access-6k7lx\") pod \"49f8efb7-37ed-4442-8959-e7b41071ae27\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") "
Jul 28 13:08:11.842510 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.842416    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-cabundle-cert\") pod \"49f8efb7-37ed-4442-8959-e7b41071ae27\" (UID: \"49f8efb7-37ed-4442-8959-e7b41071ae27\") "
Jul 28 13:08:11.842675 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.842512    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f8efb7-37ed-4442-8959-e7b41071ae27-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49f8efb7-37ed-4442-8959-e7b41071ae27" (UID: "49f8efb7-37ed-4442-8959-e7b41071ae27"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:08:11.842675 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.842623    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49f8efb7-37ed-4442-8959-e7b41071ae27" (UID: "49f8efb7-37ed-4442-8959-e7b41071ae27"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:11.842836 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.842800    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "49f8efb7-37ed-4442-8959-e7b41071ae27" (UID: "49f8efb7-37ed-4442-8959-e7b41071ae27"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:11.842836 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.842827    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:08:11.843060 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.842847    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49f8efb7-37ed-4442-8959-e7b41071ae27-kserve-provision-location\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:08:11.844468 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.844443    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f8efb7-37ed-4442-8959-e7b41071ae27-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49f8efb7-37ed-4442-8959-e7b41071ae27" (UID: "49f8efb7-37ed-4442-8959-e7b41071ae27"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:08:11.844468 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.844455    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-api-access-6k7lx" (OuterVolumeSpecName: "kube-api-access-6k7lx") pod "49f8efb7-37ed-4442-8959-e7b41071ae27" (UID: "49f8efb7-37ed-4442-8959-e7b41071ae27"). InnerVolumeSpecName "kube-api-access-6k7lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:08:11.944013 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.943904    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49f8efb7-37ed-4442-8959-e7b41071ae27-cabundle-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:08:11.944013 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.943945    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49f8efb7-37ed-4442-8959-e7b41071ae27-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:08:11.944013 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:11.943955    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k7lx\" (UniqueName: \"kubernetes.io/projected/49f8efb7-37ed-4442-8959-e7b41071ae27-kube-api-access-6k7lx\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:08:12.575581 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:12.575502    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4e2983-predictor-79ccd5778-ntk66_49f8efb7-37ed-4442-8959-e7b41071ae27/storage-initializer/1.log"
Jul 28 13:08:12.575926 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:12.575592    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"
Jul 28 13:08:12.575926 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:12.575591    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66" event={"ID":"49f8efb7-37ed-4442-8959-e7b41071ae27","Type":"ContainerDied","Data":"b00a0942499a6552ed50948851ede4634457034aa3010cb770c42f2473bd194e"}
Jul 28 13:08:12.575926 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:12.575741    2569 scope.go:117] "RemoveContainer" containerID="fa57a2de6d21a8e3ea0222be57f2c7442c422271c264f1e4b22739844eb80934"
Jul 28 13:08:12.611904 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:12.611878    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"]
Jul 28 13:08:12.616559 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:12.616535    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4e2983-predictor-79ccd5778-ntk66"]
Jul 28 13:08:14.251581 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:14.251503    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49f8efb7-37ed-4442-8959-e7b41071ae27" path="/var/lib/kubelet/pods/49f8efb7-37ed-4442-8959-e7b41071ae27/volumes"
Jul 28 13:08:16.653475 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.653442    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"]
Jul 28 13:08:16.653919 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.653848    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f8efb7-37ed-4442-8959-e7b41071ae27" containerName="storage-initializer"
Jul 28 13:08:16.653919 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.653861    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f8efb7-37ed-4442-8959-e7b41071ae27" containerName="storage-initializer"
Jul 28 13:08:16.654063 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.653935    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f8efb7-37ed-4442-8959-e7b41071ae27" containerName="storage-initializer"
Jul 28 13:08:16.654063 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.653944    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f8efb7-37ed-4442-8959-e7b41071ae27" containerName="storage-initializer"
Jul 28 13:08:16.654063 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.654023    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f8efb7-37ed-4442-8959-e7b41071ae27" containerName="storage-initializer"
Jul 28 13:08:16.654063 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.654029    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f8efb7-37ed-4442-8959-e7b41071ae27" containerName="storage-initializer"
Jul 28 13:08:16.658693 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.658675    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.661021 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.661002    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-d7907-predictor-serving-cert\""
Jul 28 13:08:16.661171 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.661154    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:08:16.661218 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.661205    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:08:16.661554 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.661535    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:08:16.661662 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.661633    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-d7907-kube-rbac-proxy-sar-config\""
Jul 28 13:08:16.661716 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.661701    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rtw6d\""
Jul 28 13:08:16.662274 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.662259    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:08:16.667299 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.667281    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"]
Jul 28 13:08:16.781718 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.781681    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2855480f-b55d-4252-8358-7beec6f605bc-proxy-tls\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.781891 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.781748    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2855480f-b55d-4252-8358-7beec6f605bc-kserve-provision-location\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.781891 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.781782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-cabundle-cert\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.781891 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.781804    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.781891 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.781828    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2sxc\" (UniqueName: \"kubernetes.io/projected/2855480f-b55d-4252-8358-7beec6f605bc-kube-api-access-b2sxc\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.883068 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.883018    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2855480f-b55d-4252-8358-7beec6f605bc-proxy-tls\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.883219 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.883120    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2855480f-b55d-4252-8358-7beec6f605bc-kserve-provision-location\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.883219 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.883160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-cabundle-cert\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.883219 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:08:16.883172    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-d7907-predictor-serving-cert: secret "raw-sklearn-d7907-predictor-serving-cert" not found
Jul 28 13:08:16.883219 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.883190    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.883442 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.883220    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2sxc\" (UniqueName: \"kubernetes.io/projected/2855480f-b55d-4252-8358-7beec6f605bc-kube-api-access-b2sxc\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.883442 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:08:16.883246    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2855480f-b55d-4252-8358-7beec6f605bc-proxy-tls podName:2855480f-b55d-4252-8358-7beec6f605bc nodeName:}" failed. No retries permitted until 2026-07-28 13:08:17.383229359 +0000 UTC m=+2065.598745992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2855480f-b55d-4252-8358-7beec6f605bc-proxy-tls") pod "raw-sklearn-d7907-predictor-766dd87b76-268ht" (UID: "2855480f-b55d-4252-8358-7beec6f605bc") : secret "raw-sklearn-d7907-predictor-serving-cert" not found
Jul 28 13:08:16.883617 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.883593    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2855480f-b55d-4252-8358-7beec6f605bc-kserve-provision-location\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.883887 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.883871    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-cabundle-cert\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.883930 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.883876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:16.893793 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:16.893767    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2sxc\" (UniqueName: \"kubernetes.io/projected/2855480f-b55d-4252-8358-7beec6f605bc-kube-api-access-b2sxc\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:17.388771 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:17.388736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2855480f-b55d-4252-8358-7beec6f605bc-proxy-tls\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:17.391160 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:17.391129    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2855480f-b55d-4252-8358-7beec6f605bc-proxy-tls\") pod \"raw-sklearn-d7907-predictor-766dd87b76-268ht\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") " pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:17.570802 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:17.570765    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:17.703675 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:17.703648    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"]
Jul 28 13:08:17.705533 ip-10-0-142-139 kubenswrapper[2569]: W0728 13:08:17.705501    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2855480f_b55d_4252_8358_7beec6f605bc.slice/crio-01575e14b2c969317481d2e5486afeac6e13b78aae0b18b13b8e143c830ad32c WatchSource:0}: Error finding container 01575e14b2c969317481d2e5486afeac6e13b78aae0b18b13b8e143c830ad32c: Status 404 returned error can't find the container with id 01575e14b2c969317481d2e5486afeac6e13b78aae0b18b13b8e143c830ad32c
Jul 28 13:08:18.608765 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:18.608729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" event={"ID":"2855480f-b55d-4252-8358-7beec6f605bc","Type":"ContainerStarted","Data":"8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c"}
Jul 28 13:08:18.608971 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:18.608788    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" event={"ID":"2855480f-b55d-4252-8358-7beec6f605bc","Type":"ContainerStarted","Data":"01575e14b2c969317481d2e5486afeac6e13b78aae0b18b13b8e143c830ad32c"}
Jul 28 13:08:21.625135 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:21.625098    2569 generic.go:356] "Generic (PLEG): container finished" podID="2855480f-b55d-4252-8358-7beec6f605bc" containerID="8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c" exitCode=0
Jul 28 13:08:21.625609 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:21.625152    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" event={"ID":"2855480f-b55d-4252-8358-7beec6f605bc","Type":"ContainerDied","Data":"8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c"}
Jul 28 13:08:22.631165 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:22.631126    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" event={"ID":"2855480f-b55d-4252-8358-7beec6f605bc","Type":"ContainerStarted","Data":"722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b"}
Jul 28 13:08:22.631605 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:22.631193    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" event={"ID":"2855480f-b55d-4252-8358-7beec6f605bc","Type":"ContainerStarted","Data":"4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394"}
Jul 28 13:08:22.631605 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:22.631390    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:22.652802 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:22.652750    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podStartSLOduration=2.935532899 podStartE2EDuration="6.652736693s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717203794s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:08:22.650965278 +0000 UTC m=+2070.866481957" watchObservedRunningTime="2026-07-28 13:08:22.652736693 +0000 UTC m=+2070.868253345"
Jul 28 13:08:23.635536 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:23.635502    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:23.636819 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:23.636791    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:08:24.640421 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:24.640379    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:08:29.645418 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:29.645389    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:08:29.645927 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:29.645896    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:08:39.646249 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:39.646204    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:08:49.646196 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:49.646154    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:08:59.645905 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:08:59.645865    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:09:09.646728 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:09.646679    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:09:19.646627 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:19.646585    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:09:29.646420 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:29.646381    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:09:39.647162 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:39.647131    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:09:46.784378 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:46.784290    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"]
Jul 28 13:09:46.784905 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:46.784558    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" containerID="cri-o://4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394" gracePeriod=30
Jul 28 13:09:46.784905 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:46.784586    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kube-rbac-proxy" containerID="cri-o://722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b" gracePeriod=30
Jul 28 13:09:47.008663 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:47.008628    2569 generic.go:356] "Generic (PLEG): container finished" podID="2855480f-b55d-4252-8358-7beec6f605bc" containerID="722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b" exitCode=2
Jul 28 13:09:47.008845 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:47.008705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" event={"ID":"2855480f-b55d-4252-8358-7beec6f605bc","Type":"ContainerDied","Data":"722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b"}
Jul 28 13:09:49.640709 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:49.640657    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 28 13:09:49.645956 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:49.645921    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:09:51.439618 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.439594    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:09:51.610664 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.610572    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-cabundle-cert\") pod \"2855480f-b55d-4252-8358-7beec6f605bc\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") "
Jul 28 13:09:51.610664 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.610651    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-kube-rbac-proxy-sar-config\") pod \"2855480f-b55d-4252-8358-7beec6f605bc\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") "
Jul 28 13:09:51.610857 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.610683    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2855480f-b55d-4252-8358-7beec6f605bc-proxy-tls\") pod \"2855480f-b55d-4252-8358-7beec6f605bc\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") "
Jul 28 13:09:51.610857 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.610785    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2sxc\" (UniqueName: \"kubernetes.io/projected/2855480f-b55d-4252-8358-7beec6f605bc-kube-api-access-b2sxc\") pod \"2855480f-b55d-4252-8358-7beec6f605bc\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") "
Jul 28 13:09:51.610968 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.610873    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2855480f-b55d-4252-8358-7beec6f605bc-kserve-provision-location\") pod \"2855480f-b55d-4252-8358-7beec6f605bc\" (UID: \"2855480f-b55d-4252-8358-7beec6f605bc\") "
Jul 28 13:09:51.611054 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.611037    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2855480f-b55d-4252-8358-7beec6f605bc" (UID: "2855480f-b55d-4252-8358-7beec6f605bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:09:51.611111 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.611046    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2855480f-b55d-4252-8358-7beec6f605bc" (UID: "2855480f-b55d-4252-8358-7beec6f605bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:09:51.611199 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.611174    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:09:51.611199 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.611187    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2855480f-b55d-4252-8358-7beec6f605bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2855480f-b55d-4252-8358-7beec6f605bc" (UID: "2855480f-b55d-4252-8358-7beec6f605bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:09:51.611381 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.611196    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2855480f-b55d-4252-8358-7beec6f605bc-cabundle-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:09:51.613298 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.613270    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2855480f-b55d-4252-8358-7beec6f605bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2855480f-b55d-4252-8358-7beec6f605bc" (UID: "2855480f-b55d-4252-8358-7beec6f605bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:09:51.613298 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.613282    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2855480f-b55d-4252-8358-7beec6f605bc-kube-api-access-b2sxc" (OuterVolumeSpecName: "kube-api-access-b2sxc") pod "2855480f-b55d-4252-8358-7beec6f605bc" (UID: "2855480f-b55d-4252-8358-7beec6f605bc"). InnerVolumeSpecName "kube-api-access-b2sxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:09:51.711721 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.711682    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2855480f-b55d-4252-8358-7beec6f605bc-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:09:51.711721 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.711715    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2sxc\" (UniqueName: \"kubernetes.io/projected/2855480f-b55d-4252-8358-7beec6f605bc-kube-api-access-b2sxc\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:09:51.711721 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.711725    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2855480f-b55d-4252-8358-7beec6f605bc-kserve-provision-location\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:09:51.973671 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.973632    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"]
Jul 28 13:09:51.974013 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.974001    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="storage-initializer"
Jul 28 13:09:51.974077 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.974014    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="storage-initializer"
Jul 28 13:09:51.974077 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.974026    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kube-rbac-proxy"
Jul 28 13:09:51.974077 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.974031    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kube-rbac-proxy"
Jul 28 13:09:51.974077 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.974045    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container"
Jul 28 13:09:51.974077 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.974051    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container"
Jul 28 13:09:51.974265 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.974154    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kserve-container"
Jul 28 13:09:51.974265 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.974169    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2855480f-b55d-4252-8358-7beec6f605bc" containerName="kube-rbac-proxy"
Jul 28 13:09:51.977601 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.977580    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:51.980155 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.980136    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-cfc8f-kube-rbac-proxy-sar-config\""
Jul 28 13:09:51.980254 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.980136    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-cfc8f-predictor-serving-cert\""
Jul 28 13:09:51.989441 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:51.989412    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"]
Jul 28 13:09:52.033187 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.033155    2569 generic.go:356] "Generic (PLEG): container finished" podID="2855480f-b55d-4252-8358-7beec6f605bc" containerID="4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394" exitCode=0
Jul 28 13:09:52.033364 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.033208    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"
Jul 28 13:09:52.033364 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.033218    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" event={"ID":"2855480f-b55d-4252-8358-7beec6f605bc","Type":"ContainerDied","Data":"4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394"}
Jul 28 13:09:52.033364 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.033266    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht" event={"ID":"2855480f-b55d-4252-8358-7beec6f605bc","Type":"ContainerDied","Data":"01575e14b2c969317481d2e5486afeac6e13b78aae0b18b13b8e143c830ad32c"}
Jul 28 13:09:52.033364 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.033293    2569 scope.go:117] "RemoveContainer" containerID="722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b"
Jul 28 13:09:52.042237 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.042218    2569 scope.go:117] "RemoveContainer" containerID="4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394"
Jul 28 13:09:52.050058 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.050040    2569 scope.go:117] "RemoveContainer" containerID="8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c"
Jul 28 13:09:52.057040 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.057011    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"]
Jul 28 13:09:52.058893 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.058877    2569 scope.go:117] "RemoveContainer" containerID="722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b"
Jul 28 13:09:52.059229 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:09:52.059208    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b\": container with ID starting with 722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b not found: ID does not exist" containerID="722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b"
Jul 28 13:09:52.059313 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.059238    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b"} err="failed to get container status \"722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b\": rpc error: code = NotFound desc = could not find container \"722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b\": container with ID starting with 722c3df987fdfc28e62e487737167808e49a4bbc1ad3eb4b073002b62d23756b not found: ID does not exist"
Jul 28 13:09:52.059313 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.059258    2569 scope.go:117] "RemoveContainer" containerID="4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394"
Jul 28 13:09:52.059573 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:09:52.059545    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394\": container with ID starting with 4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394 not found: ID does not exist" containerID="4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394"
Jul 28 13:09:52.059624 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.059584    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394"} err="failed to get container status \"4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394\": rpc error: code = NotFound desc = could not find container \"4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394\": container with ID starting with 4fbe302c8a8fdaa4355d9fbc602d818af5534d2a582a19daf214db3906d9d394 not found: ID does not exist"
Jul 28 13:09:52.059624 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.059607    2569 scope.go:117] "RemoveContainer" containerID="8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c"
Jul 28 13:09:52.059866 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:09:52.059847    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c\": container with ID starting with 8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c not found: ID does not exist" containerID="8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c"
Jul 28 13:09:52.059919 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.059870    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c"} err="failed to get container status \"8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c\": rpc error: code = NotFound desc = could not find container \"8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c\": container with ID starting with 8ef766a74ebb8ffb86be14b8ab5e8b66e4c5d8ba0349e8f8d5423229aba4ae3c not found: ID does not exist"
Jul 28 13:09:52.059961 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.059935    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d7907-predictor-766dd87b76-268ht"]
Jul 28 13:09:52.115223 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.115193    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22vl9\" (UniqueName: \"kubernetes.io/projected/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-api-access-22vl9\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.115395 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.115249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01eb2003-620d-45b7-a833-1dabb9b421f4-proxy-tls\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.115395 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.115302    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01eb2003-620d-45b7-a833-1dabb9b421f4-kserve-provision-location\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.115395 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.115334    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-cabundle-cert\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.115395 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.115365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.216406 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.216368    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22vl9\" (UniqueName: \"kubernetes.io/projected/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-api-access-22vl9\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.216535 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.216434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01eb2003-620d-45b7-a833-1dabb9b421f4-proxy-tls\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.216535 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.216457    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01eb2003-620d-45b7-a833-1dabb9b421f4-kserve-provision-location\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.216535 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.216485    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-cabundle-cert\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.216704 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.216613    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.216889 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.216869    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01eb2003-620d-45b7-a833-1dabb9b421f4-kserve-provision-location\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.217134 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.217118    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-cabundle-cert\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.219370 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.219354    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-cfc8f-kube-rbac-proxy-sar-config\""
Jul 28 13:09:52.219428 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.219387    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-cfc8f-predictor-serving-cert\""
Jul 28 13:09:52.225668 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.225609    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22vl9\" (UniqueName: \"kubernetes.io/projected/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-api-access-22vl9\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.227848 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.227826    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.229326 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.229306    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01eb2003-620d-45b7-a833-1dabb9b421f4-proxy-tls\") pod \"raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.254974 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.254943    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2855480f-b55d-4252-8358-7beec6f605bc" path="/var/lib/kubelet/pods/2855480f-b55d-4252-8358-7beec6f605bc/volumes"
Jul 28 13:09:52.288208 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.288171    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:52.427079 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:52.427046    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"]
Jul 28 13:09:52.433233 ip-10-0-142-139 kubenswrapper[2569]: W0728 13:09:52.431105    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01eb2003_620d_45b7_a833_1dabb9b421f4.slice/crio-2740fca4cf14c1c96270e94f8710904b83354814932320af833c32617efae422 WatchSource:0}: Error finding container 2740fca4cf14c1c96270e94f8710904b83354814932320af833c32617efae422: Status 404 returned error can't find the container with id 2740fca4cf14c1c96270e94f8710904b83354814932320af833c32617efae422
Jul 28 13:09:53.039281 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:53.039247    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" event={"ID":"01eb2003-620d-45b7-a833-1dabb9b421f4","Type":"ContainerStarted","Data":"9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9"}
Jul 28 13:09:53.039692 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:53.039309    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" event={"ID":"01eb2003-620d-45b7-a833-1dabb9b421f4","Type":"ContainerStarted","Data":"2740fca4cf14c1c96270e94f8710904b83354814932320af833c32617efae422"}
Jul 28 13:09:57.059154 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:57.059118    2569 generic.go:356] "Generic (PLEG): container finished" podID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerID="9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9" exitCode=0
Jul 28 13:09:57.059542 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:57.059189    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" event={"ID":"01eb2003-620d-45b7-a833-1dabb9b421f4","Type":"ContainerDied","Data":"9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9"}
Jul 28 13:09:58.065656 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:58.065622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" event={"ID":"01eb2003-620d-45b7-a833-1dabb9b421f4","Type":"ContainerStarted","Data":"eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466"}
Jul 28 13:09:58.066038 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:58.065683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" event={"ID":"01eb2003-620d-45b7-a833-1dabb9b421f4","Type":"ContainerStarted","Data":"daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1"}
Jul 28 13:09:58.066038 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:58.065979    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:58.066155 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:58.066138    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:09:58.067765 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:58.067732    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:09:58.087856 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:58.087802    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podStartSLOduration=3.319974127 podStartE2EDuration="7.087787104s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.767812977s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:09:58.085956065 +0000 UTC m=+2166.301472740" watchObservedRunningTime="2026-07-28 13:09:58.087787104 +0000 UTC m=+2166.303303756"
Jul 28 13:09:59.070897 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:09:59.070859    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:10:04.076477 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:10:04.076446    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:10:04.077154 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:10:04.077122    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:10:14.076980 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:10:14.076942    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:10:24.077004 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:10:24.076960    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:10:34.077556 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:10:34.077512    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:10:44.077121 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:10:44.077076    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:10:54.077905 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:10:54.077865    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:11:04.077513 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:04.077469    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:11:10.251517 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:10.251489    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:11:12.074806 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:12.074774    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"]
Jul 28 13:11:12.075214 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:12.074977    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container" containerID="cri-o://daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1" gracePeriod=30
Jul 28 13:11:12.075214 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:12.075035    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kube-rbac-proxy" containerID="cri-o://eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466" gracePeriod=30
Jul 28 13:11:12.397640 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:12.397560    2569 generic.go:356] "Generic (PLEG): container finished" podID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerID="eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466" exitCode=2
Jul 28 13:11:12.397640 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:12.397597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" event={"ID":"01eb2003-620d-45b7-a833-1dabb9b421f4","Type":"ContainerDied","Data":"eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466"}
Jul 28 13:11:14.071881 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:14.071835    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 28 13:11:16.720806 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.720782    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:11:16.849661 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.849572    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-cabundle-cert\") pod \"01eb2003-620d-45b7-a833-1dabb9b421f4\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") "
Jul 28 13:11:16.849661 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.849634    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22vl9\" (UniqueName: \"kubernetes.io/projected/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-api-access-22vl9\") pod \"01eb2003-620d-45b7-a833-1dabb9b421f4\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") "
Jul 28 13:11:16.849661 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.849655    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-rbac-proxy-sar-config\") pod \"01eb2003-620d-45b7-a833-1dabb9b421f4\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") "
Jul 28 13:11:16.849892 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.849690    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01eb2003-620d-45b7-a833-1dabb9b421f4-proxy-tls\") pod \"01eb2003-620d-45b7-a833-1dabb9b421f4\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") "
Jul 28 13:11:16.849892 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.849732    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01eb2003-620d-45b7-a833-1dabb9b421f4-kserve-provision-location\") pod \"01eb2003-620d-45b7-a833-1dabb9b421f4\" (UID: \"01eb2003-620d-45b7-a833-1dabb9b421f4\") "
Jul 28 13:11:16.850158 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.850124    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01eb2003-620d-45b7-a833-1dabb9b421f4" (UID: "01eb2003-620d-45b7-a833-1dabb9b421f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:11:16.850264 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.850149    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01eb2003-620d-45b7-a833-1dabb9b421f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01eb2003-620d-45b7-a833-1dabb9b421f4" (UID: "01eb2003-620d-45b7-a833-1dabb9b421f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:11:16.850264 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.850128    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01eb2003-620d-45b7-a833-1dabb9b421f4" (UID: "01eb2003-620d-45b7-a833-1dabb9b421f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:11:16.851824 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.851791    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01eb2003-620d-45b7-a833-1dabb9b421f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01eb2003-620d-45b7-a833-1dabb9b421f4" (UID: "01eb2003-620d-45b7-a833-1dabb9b421f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:11:16.851930 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.851882    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-api-access-22vl9" (OuterVolumeSpecName: "kube-api-access-22vl9") pod "01eb2003-620d-45b7-a833-1dabb9b421f4" (UID: "01eb2003-620d-45b7-a833-1dabb9b421f4"). InnerVolumeSpecName "kube-api-access-22vl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:11:16.950589 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.950548    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22vl9\" (UniqueName: \"kubernetes.io/projected/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-api-access-22vl9\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:11:16.950589 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.950584    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:11:16.950589 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.950593    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01eb2003-620d-45b7-a833-1dabb9b421f4-proxy-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:11:16.950825 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.950603    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01eb2003-620d-45b7-a833-1dabb9b421f4-kserve-provision-location\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:11:16.950825 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:16.950613    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01eb2003-620d-45b7-a833-1dabb9b421f4-cabundle-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\""
Jul 28 13:11:17.428553 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.428511    2569 generic.go:356] "Generic (PLEG): container finished" podID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerID="daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1" exitCode=0
Jul 28 13:11:17.428728 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.428578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" event={"ID":"01eb2003-620d-45b7-a833-1dabb9b421f4","Type":"ContainerDied","Data":"daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1"}
Jul 28 13:11:17.428728 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.428589    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"
Jul 28 13:11:17.428728 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.428668    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws" event={"ID":"01eb2003-620d-45b7-a833-1dabb9b421f4","Type":"ContainerDied","Data":"2740fca4cf14c1c96270e94f8710904b83354814932320af833c32617efae422"}
Jul 28 13:11:17.428728 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.428707    2569 scope.go:117] "RemoveContainer" containerID="eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466"
Jul 28 13:11:17.437950 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.437921    2569 scope.go:117] "RemoveContainer" containerID="daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1"
Jul 28 13:11:17.445570 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.445551    2569 scope.go:117] "RemoveContainer" containerID="9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9"
Jul 28 13:11:17.452524 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.452495    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"]
Jul 28 13:11:17.454426 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.454403    2569 scope.go:117] "RemoveContainer" containerID="eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466"
Jul 28 13:11:17.454684 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:11:17.454664    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466\": container with ID starting with eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466 not found: ID does not exist" containerID="eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466"
Jul 28 13:11:17.454749 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.454694    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466"} err="failed to get container status \"eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466\": rpc error: code = NotFound desc = could not find container \"eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466\": container with ID starting with eebc9da352bf1c17c775b19d2e5afc892038711b9dab1e4323091ca63f5d9466 not found: ID does not exist"
Jul 28 13:11:17.454749 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.454710    2569 scope.go:117] "RemoveContainer" containerID="daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1"
Jul 28 13:11:17.454953 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:11:17.454933    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1\": container with ID starting with daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1 not found: ID does not exist" containerID="daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1"
Jul 28 13:11:17.455045 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.454962    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1"} err="failed to get container status \"daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1\": rpc error: code = NotFound desc = could not find container \"daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1\": container with ID starting with daa5f1c0a7dcafbf9b6e9c540ebc5ed74e657ae643b528c943fc5ff7f65903a1 not found: ID does not exist"
Jul 28 13:11:17.455045 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.455006    2569 scope.go:117] "RemoveContainer" containerID="9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9"
Jul 28 13:11:17.455266 ip-10-0-142-139 kubenswrapper[2569]: E0728 13:11:17.455250    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9\": container with ID starting with 9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9 not found: ID does not exist" containerID="9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9"
Jul 28 13:11:17.455312 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.455269    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9"} err="failed to get container status \"9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9\": rpc error: code = NotFound desc = could not find container \"9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9\": container with ID starting with 9693027864d51db38a01b3869d36c5d961b7cbda76dfa4dcef61ff297c838fa9 not found: ID does not exist"
Jul 28 13:11:17.456033 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:17.455972    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfc8f-predictor-7479f9b668-qj5ws"]
Jul 28 13:11:18.251737 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:18.251701    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" path="/var/lib/kubelet/pods/01eb2003-620d-45b7-a833-1dabb9b421f4/volumes"
Jul 28 13:11:52.237932 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:52.237897    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xdsfj_79d7959e-9636-495a-930c-a09f71fb6c57/global-pull-secret-syncer/0.log"
Jul 28 13:11:52.392477 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:52.392449    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lcl5q_fbe4ae7f-5807-4215-8fc4-a708450f9777/konnectivity-agent/0.log"
Jul 28 13:11:52.520677 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:52.520599    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-139.ec2.internal_19f6277a0055c64139d8e6ee0d720a13/haproxy/0.log"
Jul 28 13:11:56.330768 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.330736    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7465f793-3b4e-43d5-92c9-a538ed297a7b/alertmanager/0.log"
Jul 28 13:11:56.359580 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.359547    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7465f793-3b4e-43d5-92c9-a538ed297a7b/config-reloader/0.log"
Jul 28 13:11:56.387161 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.387128    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7465f793-3b4e-43d5-92c9-a538ed297a7b/kube-rbac-proxy-web/0.log"
Jul 28 13:11:56.417177 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.417154    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7465f793-3b4e-43d5-92c9-a538ed297a7b/kube-rbac-proxy/0.log"
Jul 28 13:11:56.443014 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.442978    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7465f793-3b4e-43d5-92c9-a538ed297a7b/kube-rbac-proxy-metric/0.log"
Jul 28 13:11:56.471677 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.471651    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7465f793-3b4e-43d5-92c9-a538ed297a7b/prom-label-proxy/0.log"
Jul 28 13:11:56.497612 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.497587    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7465f793-3b4e-43d5-92c9-a538ed297a7b/init-config-reloader/0.log"
Jul 28 13:11:56.588648 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.588568    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-g5dt2_845b2466-b87c-4d92-97ba-0f2309e87624/kube-state-metrics/0.log"
Jul 28 13:11:56.612654 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.612626    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-g5dt2_845b2466-b87c-4d92-97ba-0f2309e87624/kube-rbac-proxy-main/0.log"
Jul 28 13:11:56.645421 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.645389    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-g5dt2_845b2466-b87c-4d92-97ba-0f2309e87624/kube-rbac-proxy-self/0.log"
Jul 28 13:11:56.681811 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.681785    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6759974dfc-8ghkv_5773d6fb-afa0-4db1-8105-ce4d08e50a59/metrics-server/0.log"
Jul 28 13:11:56.716770 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.716739    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-m2czb_274ac08e-e89c-47f6-8236-34e87fd7f597/monitoring-plugin/0.log"
Jul 28 13:11:56.863619 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.863539    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbx9v_846183c6-9edf-4a5e-b6ee-de3752e60c06/node-exporter/0.log"
Jul 28 13:11:56.895501 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.895477    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbx9v_846183c6-9edf-4a5e-b6ee-de3752e60c06/kube-rbac-proxy/0.log"
Jul 28 13:11:56.922431 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:56.922410    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbx9v_846183c6-9edf-4a5e-b6ee-de3752e60c06/init-textfile/0.log"
Jul 28 13:11:57.073628 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.073594    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-j6bx6_dc80c0af-bbf0-43f8-a9fc-f72af0510f41/kube-rbac-proxy-main/0.log"
Jul 28 13:11:57.098591 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.098556    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-j6bx6_dc80c0af-bbf0-43f8-a9fc-f72af0510f41/kube-rbac-proxy-self/0.log"
Jul 28 13:11:57.128838 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.128752    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-j6bx6_dc80c0af-bbf0-43f8-a9fc-f72af0510f41/openshift-state-metrics/0.log"
Jul 28 13:11:57.424354 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.424279    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-q8qms_42193b4e-1fdd-45fb-9e27-91533f3b564f/prometheus-operator/0.log"
Jul 28 13:11:57.449350 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.449317    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-q8qms_42193b4e-1fdd-45fb-9e27-91533f3b564f/kube-rbac-proxy/0.log"
Jul 28 13:11:57.513961 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.513929    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-765d778646-7ws2w_7b7742da-776d-44e7-8004-83c4d730e8cf/telemeter-client/0.log"
Jul 28 13:11:57.543838 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.543805    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-765d778646-7ws2w_7b7742da-776d-44e7-8004-83c4d730e8cf/reload/0.log"
Jul 28 13:11:57.570159 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.570124    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-765d778646-7ws2w_7b7742da-776d-44e7-8004-83c4d730e8cf/kube-rbac-proxy/0.log"
Jul 28 13:11:57.609584 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.609559    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698d9cc9fc-6f874_9b774a01-b5c9-42a9-bc44-e2237cf95830/thanos-query/0.log"
Jul 28 13:11:57.647266 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.647242    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698d9cc9fc-6f874_9b774a01-b5c9-42a9-bc44-e2237cf95830/kube-rbac-proxy-web/0.log"
Jul 28 13:11:57.677738 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.677669    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698d9cc9fc-6f874_9b774a01-b5c9-42a9-bc44-e2237cf95830/kube-rbac-proxy/0.log"
Jul 28 13:11:57.717033 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.717006    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698d9cc9fc-6f874_9b774a01-b5c9-42a9-bc44-e2237cf95830/prom-label-proxy/0.log"
Jul 28 13:11:57.751348 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.751320    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698d9cc9fc-6f874_9b774a01-b5c9-42a9-bc44-e2237cf95830/kube-rbac-proxy-rules/0.log"
Jul 28 13:11:57.782557 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:57.782530    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698d9cc9fc-6f874_9b774a01-b5c9-42a9-bc44-e2237cf95830/kube-rbac-proxy-metrics/0.log"
Jul 28 13:11:59.767363 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.767322    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"]
Jul 28 13:11:59.767764 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.767675    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container"
Jul 28 13:11:59.767764 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.767686    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container"
Jul 28 13:11:59.767764 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.767705    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="storage-initializer"
Jul 28 13:11:59.767764 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.767712    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="storage-initializer"
Jul 28 13:11:59.767764 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.767720    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kube-rbac-proxy"
Jul 28 13:11:59.767764 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.767725    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kube-rbac-proxy"
Jul 28 13:11:59.768030 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.767790    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kserve-container"
Jul 28 13:11:59.768030 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.767798    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="01eb2003-620d-45b7-a833-1dabb9b421f4" containerName="kube-rbac-proxy"
Jul 28 13:11:59.770776 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.770759    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.773212 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.773193    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4r45x\"/\"default-dockercfg-8kmjh\""
Jul 28 13:11:59.773302 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.773193    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4r45x\"/\"kube-root-ca.crt\""
Jul 28 13:11:59.773941 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.773927    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4r45x\"/\"openshift-service-ca.crt\""
Jul 28 13:11:59.779788 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.779721    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"]
Jul 28 13:11:59.804208 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.804176    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-lib-modules\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.804375 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.804230    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-proc\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.804375 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.804292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-podres\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.804375 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.804321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-sys\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.804375 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.804347    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts9f5\" (UniqueName: \"kubernetes.io/projected/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-kube-api-access-ts9f5\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.905812 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.905775    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-proc\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.906013 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.905828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-podres\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.906013 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.905872    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-sys\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.906013 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.905906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts9f5\" (UniqueName: \"kubernetes.io/projected/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-kube-api-access-ts9f5\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.906013 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.905959    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-lib-modules\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.906199 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.905907    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-proc\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.906199 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.906016    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-sys\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.906199 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.906041    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-podres\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.906199 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.906143    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-lib-modules\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.915144 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.915124    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts9f5\" (UniqueName: \"kubernetes.io/projected/e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab-kube-api-access-ts9f5\") pod \"perf-node-gather-daemonset-2v9v6\" (UID: \"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab\") " pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:11:59.942555 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:11:59.942529    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-847fd8d496-twwm9_0b4a497c-bda8-4a7c-8418-ac7dd8d27ed1/console/0.log"
Jul 28 13:12:00.081983 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:00.081890    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:12:00.217098 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:00.217068    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"]
Jul 28 13:12:00.219560 ip-10-0-142-139 kubenswrapper[2569]: W0728 13:12:00.219529    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode49ac7bb_f72c_4787_bdf3_84d4b9a4dcab.slice/crio-41a363a9d9b5df31250f0860d40c53da7c1217608b28b7bdc833918036212bce WatchSource:0}: Error finding container 41a363a9d9b5df31250f0860d40c53da7c1217608b28b7bdc833918036212bce: Status 404 returned error can't find the container with id 41a363a9d9b5df31250f0860d40c53da7c1217608b28b7bdc833918036212bce
Jul 28 13:12:00.620406 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:00.620371    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6" event={"ID":"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab","Type":"ContainerStarted","Data":"187ed3ce0af18cbb682ed50a0da9791813011fbfd510d9c7534c92748efa5b1f"}
Jul 28 13:12:00.620562 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:00.620426    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6" event={"ID":"e49ac7bb-f72c-4787-bdf3-84d4b9a4dcab","Type":"ContainerStarted","Data":"41a363a9d9b5df31250f0860d40c53da7c1217608b28b7bdc833918036212bce"}
Jul 28 13:12:00.620562 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:00.620458    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:12:00.637953 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:00.637898    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6" podStartSLOduration=1.6378822830000002 podStartE2EDuration="1.637882283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:12:00.636305506 +0000 UTC m=+2288.851822158" watchObservedRunningTime="2026-07-28 13:12:00.637882283 +0000 UTC m=+2288.853398935"
Jul 28 13:12:01.436629 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:01.436582    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hwfbj_13ffc097-2363-4b74-a9d8-f6865e6f22dc/dns/0.log"
Jul 28 13:12:01.460843 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:01.460821    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hwfbj_13ffc097-2363-4b74-a9d8-f6865e6f22dc/kube-rbac-proxy/0.log"
Jul 28 13:12:01.598954 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:01.598920    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zkstz_793c18e4-9309-4da3-956e-5791d83891cf/dns-node-resolver/0.log"
Jul 28 13:12:02.056855 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:02.056815    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-dc9fbdd49-tlf2w_8f33c0ba-7ac0-4906-823e-8632e6008279/registry/0.log"
Jul 28 13:12:02.109444 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:02.109413    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4vq25_c599f721-e2df-422a-bf36-6319e85593d2/node-ca/0.log"
Jul 28 13:12:03.291095 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:03.291050    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8bmdm_62ad7a92-64da-4091-8293-399480862c3e/serve-healthcheck-canary/0.log"
Jul 28 13:12:03.784358 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:03.784330    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f58jt_560ade9a-e010-41ab-9174-54a41e8d323e/kube-rbac-proxy/0.log"
Jul 28 13:12:03.810374 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:03.810345    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f58jt_560ade9a-e010-41ab-9174-54a41e8d323e/exporter/0.log"
Jul 28 13:12:03.836260 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:03.836226    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f58jt_560ade9a-e010-41ab-9174-54a41e8d323e/extractor/0.log"
Jul 28 13:12:06.233255 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:06.233209    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-f9d98_9491f529-30d3-442c-90be-082c56aa8994/manager/0.log"
Jul 28 13:12:06.255652 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:06.255624    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-qwhkq_1b1e00c1-36cf-4a6d-a96d-26e28b14b418/server/0.log"
Jul 28 13:12:06.437921 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:06.437893    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-dchcx_1b26e8e6-91ce-4cb5-b58c-a60fb3e55228/manager/0.log"
Jul 28 13:12:06.634133 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:06.634066    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4r45x/perf-node-gather-daemonset-2v9v6"
Jul 28 13:12:12.070950 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:12.070911    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7ng8z_b9de0429-8e9d-4263-8d35-b8088687e3cc/kube-multus/0.log"
Jul 28 13:12:12.309085 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:12.309051    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn8wq_0e25259e-78a2-4063-8946-e80c2954e647/kube-multus-additional-cni-plugins/0.log"
Jul 28 13:12:12.336191 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:12.336162    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn8wq_0e25259e-78a2-4063-8946-e80c2954e647/egress-router-binary-copy/0.log"
Jul 28 13:12:12.361929 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:12.361897    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn8wq_0e25259e-78a2-4063-8946-e80c2954e647/cni-plugins/0.log"
Jul 28 13:12:12.385962 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:12.385932    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn8wq_0e25259e-78a2-4063-8946-e80c2954e647/bond-cni-plugin/0.log"
Jul 28 13:12:12.415598 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:12.415571    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn8wq_0e25259e-78a2-4063-8946-e80c2954e647/routeoverride-cni/0.log"
Jul 28 13:12:12.442306 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:12.442277    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn8wq_0e25259e-78a2-4063-8946-e80c2954e647/whereabouts-cni-bincopy/0.log"
Jul 28 13:12:12.474721 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:12.474694    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn8wq_0e25259e-78a2-4063-8946-e80c2954e647/whereabouts-cni/0.log"
Jul 28 13:12:12.836935 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:12.836907    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6jp6n_afc8521c-7a71-4ec0-9e3a-53d4512d5ae3/network-metrics-daemon/0.log"
Jul 28 13:12:12.879720 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:12.879693    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6jp6n_afc8521c-7a71-4ec0-9e3a-53d4512d5ae3/kube-rbac-proxy/0.log"
Jul 28 13:12:13.773962 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:13.773930    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45gqf_69344ace-2af3-4833-bc5f-cc6e74c00933/ovn-controller/0.log"
Jul 28 13:12:13.792174 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:13.792146    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45gqf_69344ace-2af3-4833-bc5f-cc6e74c00933/ovn-acl-logging/0.log"
Jul 28 13:12:13.818302 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:13.818275    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45gqf_69344ace-2af3-4833-bc5f-cc6e74c00933/kube-rbac-proxy-node/0.log"
Jul 28 13:12:13.843579 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:13.843548    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45gqf_69344ace-2af3-4833-bc5f-cc6e74c00933/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 13:12:13.875323 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:13.875297    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45gqf_69344ace-2af3-4833-bc5f-cc6e74c00933/northd/0.log"
Jul 28 13:12:13.897428 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:13.897405    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45gqf_69344ace-2af3-4833-bc5f-cc6e74c00933/nbdb/0.log"
Jul 28 13:12:13.922136 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:13.922101    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45gqf_69344ace-2af3-4833-bc5f-cc6e74c00933/sbdb/0.log"
Jul 28 13:12:13.991031 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:13.990975    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-45gqf_69344ace-2af3-4833-bc5f-cc6e74c00933/ovnkube-controller/0.log"
Jul 28 13:12:15.540107 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:15.540078    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-kffxs_a4544365-5759-41c2-8b2a-0f96b508e3d5/network-check-target-container/0.log"
Jul 28 13:12:16.537812 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:16.537785    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-wjhph_ae3368dc-9e5f-44c0-8f49-532e8336cbb7/iptables-alerter/0.log"
Jul 28 13:12:17.255559 ip-10-0-142-139 kubenswrapper[2569]: I0728 13:12:17.255525    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-kr4z5_c533d0e1-4c5f-4cb4-8259-b83b72a39e1d/tuned/0.log"