Jul 28 20:45:11.057322 ip-10-0-138-164 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 20:45:11.350342 ip-10-0-138-164 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:45:11.350342 ip-10-0-138-164 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 20:45:11.350342 ip-10-0-138-164 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:45:11.350342 ip-10-0-138-164 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 20:45:11.350342 ip-10-0-138-164 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:45:11.352182 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.352052    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 20:45:11.355214 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355200    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:45:11.355214 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355214    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355219    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355223    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355226    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355228    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355231    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355234    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355237    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355240    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355243    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355246    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355249    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355251    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355254    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355256    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355259    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355262    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355265    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355268    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355270    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:45:11.355275 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355280    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355283    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355286    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355288    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355291    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355294    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355296    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355299    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355301    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355305    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355310    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355315    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355318    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355321    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355323    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355327    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355329    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355332    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355335    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:45:11.355751 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355337    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355339    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355342    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355344    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355347    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355349    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355351    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355354    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355357    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355359    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355362    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355364    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355367    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355369    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355373    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355376    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355379    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355382    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355385    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355387    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:45:11.356260 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355390    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355393    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355395    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355397    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355400    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355403    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355405    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355408    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355410    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355413    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355415    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355418    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355420    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355423    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355425    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355429    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355431    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355434    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355436    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355439    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:45:11.356747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355441    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355444    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355446    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355449    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355451    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355455    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355459    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355463    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355466    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355469    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355472    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355475    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355478    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355480    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355482    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355485    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355487    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355490    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.355492    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:45:11.357263 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356191    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356198    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356202    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356206    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356209    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356211    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356214    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356217    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356219    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356222    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356225    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356227    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356230    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356232    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356235    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356237    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356240    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356243    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356245    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:45:11.357713 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356248    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356251    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356254    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356259    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356263    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356266    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356269    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356272    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356274    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356277    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356280    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356283    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356286    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356288    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356292    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356295    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356297    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356300    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356302    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356305    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:45:11.358189 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356307    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356310    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356312    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356315    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356317    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356320    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356324    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356327    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356330    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356334    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356336    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356339    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356342    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356345    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356348    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356351    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356354    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356356    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356359    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356362    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:45:11.358671 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356365    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356367    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356370    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356372    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356374    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356377    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356380    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356382    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356385    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356387    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356389    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356392    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356395    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356398    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356401    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356403    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356406    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356408    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356411    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:45:11.359411 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356413    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356416    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356418    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356420    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356423    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356426    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356430    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356432    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356435    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356438    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356441    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356443    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356446    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356449    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356451    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356453    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356456    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356459    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356461    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356463    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:45:11.360104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356466    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356820    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356825    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356828    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356831    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356835    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356839    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356842    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356845    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356848    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356851    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356854    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356856    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356859    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356861    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356864    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356866    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356869    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356872    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:45:11.360623 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356875    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356877    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356881    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356886    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356889    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356892    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356894    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356897    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356900    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356902    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356905    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356907    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356910    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356912    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356915    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356917    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356920    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356923    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356926    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356929    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:45:11.361101 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356931    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356934    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356937    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356939    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356942    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356944    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356947    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356950    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356952    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356954    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356957    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356960    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356963    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356965    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356968    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356971    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356973    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356976    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356978    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356981    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:45:11.361587 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356983    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356987    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356989    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356992    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356994    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356996    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.356999    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357002    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357004    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357006    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357009    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357012    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357016    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357018    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357021    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357023    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357026    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357028    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357031    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357033    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:45:11.362278 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357035    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357038    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357040    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357044    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357046    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357049    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357051    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357054    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357057    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357059    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357062    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357064    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357066    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357069    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357071    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357089    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357091    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357094    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357096    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357099    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:45:11.362762 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.357102    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357174    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357181    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357186    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357191    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357195    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357199    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357203    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357207    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357210    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357214    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357217    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357220    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357223    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357227    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357230    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357233    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357236    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357239    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357242    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357245    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357249    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357252    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357255    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 20:45:11.363263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357259    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357262    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357265    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357269    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357272    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357275    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357278    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357281    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357284    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357288    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357291    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357293    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357296    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357299    2578 flags.go:64] FLAG: --enable-server="true"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357302    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357306    2578 flags.go:64] FLAG: --event-burst="100"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357309    2578 flags.go:64] FLAG: --event-qps="50"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357312    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357315    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357318    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357321    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357324    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357327    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357330    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357334    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 20:45:11.363904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357337    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357340    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357343    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357346    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357348    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357352    2578 flags.go:64] FLAG: --feature-gates=""
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357355    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357358    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357362    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357364    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357367    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357370    2578 flags.go:64] FLAG: --help="false"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357373    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357376    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357381    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357384    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357741    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357745    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357748    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357751    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357753    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357756    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357759    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357762    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 20:45:11.364537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357765    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357768    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357771    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357774    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357777    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357780    2578 flags.go:64] FLAG: --lock-file=""
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357783    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357786    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357789    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357794    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357797    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357800    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357802    2578 flags.go:64] FLAG: --logging-format="text"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357805    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357809    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357811    2578 flags.go:64] FLAG: --manifest-url=""
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357814    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357819    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357822    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357825    2578 flags.go:64] FLAG: --max-pods="110"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357828    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357831    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357834    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357837    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357840    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 20:45:11.365145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357843    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357846    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357853    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357856    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357859    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357862    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357864    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357869    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357872    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357875    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357877    2578 flags.go:64] FLAG: --port="10250"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357880    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357883    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04c061a4e5727f517"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357886    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357888    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357891    2578 flags.go:64] FLAG: --register-node="true"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357894    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357897    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357900    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357902    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357905    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357909    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357912    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357915    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357917    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 20:45:11.365746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357920    2578 flags.go:64] FLAG: --runonce="false"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357923    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357926    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357928    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357931    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357934    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357937    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357940    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357942    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357945    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357948    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357951    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357954    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357957    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357959    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357962    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357967    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357970    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357972    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357975    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357978    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357981    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357983    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357986    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357989    2578 flags.go:64] FLAG: --v="2"
Jul 28 20:45:11.366364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357992    2578 flags.go:64] FLAG: --version="false"
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357996    2578 flags.go:64] FLAG: --vmodule=""
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.357999    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.358002    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358122    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358126    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358130    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358132    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358135    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358137    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358140    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358142    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358145    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358147    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358151    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358153    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358156    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358158    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358161    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358163    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:45:11.366975 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358166    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358168    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358171    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358173    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358175    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358178    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358180    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358182    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358185    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358189    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358192    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358195    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358198    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358201    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358204    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358206    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358209    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358211    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358214    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358217    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:45:11.367517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358219    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358222    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358224    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358227    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358229    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358232    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358234    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358236    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358240    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358244    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358247    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358250    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358252    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358255    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358257    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358260    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358262    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358264    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358267    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358269    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:45:11.368009 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358272    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358274    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358277    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358279    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358281    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358284    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358286    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358288    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358291    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358294    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358296    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358300    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358303    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358306    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358309    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358313    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358315    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358318    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358320    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:45:11.368508 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358322    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358327    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358329    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358332    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358334    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358337    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358339    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358342    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358344    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358347    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358349    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358352    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358354    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358357    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358359    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358362    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358364    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358367    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358370    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358372    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:45:11.368968 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358375    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358377    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358380    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.358382    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.358840    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.367941    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.367956    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.367975    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.367981    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368034    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368039    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368043    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368046    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368049    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:45:11.369509 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368052    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368056    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368061    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368064    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368067    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368069    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368091    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368095    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368098    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368100    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368103    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368106    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368108    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368111    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368115    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368117    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368120    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368123    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368125    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:45:11.369865 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368128    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368130    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368133    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368135    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368138    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368141    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368144    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368146    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368149    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368152    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368154    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368156    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368159    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368162    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368164    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368167    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368169    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368171    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368174    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368176    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:45:11.370344 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368179    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368181    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368184    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368186    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368189    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368191    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368194    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368197    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368199    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368202    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368204    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368207    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368209    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368212    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368214    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368216    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368219    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368223    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368226    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368229    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:45:11.370857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368231    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368234    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368236    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368239    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368241    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368244    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368247    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368249    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368252    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368255    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368257    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368260    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368262    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368264    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368267    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368270    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368273    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368275    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368278    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368280    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:45:11.371367 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368282    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368285    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368287    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368290    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368292    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368296    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368300    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368303    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368305    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368309    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368312    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368314    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368317    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368321    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:45:11.371870 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368324    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.368329    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368441    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368446    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368449    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368452    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368455    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368458    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368460    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368463    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368466    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368469    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368471    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368475    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368478    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:45:11.372229 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368480    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368482    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368485    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368487    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368490    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368492    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368495    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368497    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368500    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368502    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368505    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368507    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368510    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368513    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368516    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368519    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368521    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368524    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368526    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368529    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:45:11.372603 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368531    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368534    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368537    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368539    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368542    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368544    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368547    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368549    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368552    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368555    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368557    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368561    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368564    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368566    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368569    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368571    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368574    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368576    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368579    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368581    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:45:11.373104 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368583    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368586    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368588    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368590    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368593    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368597    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368600    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368603    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368605    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368608    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368610    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368613    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368615    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368618    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368620    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368623    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368627    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368630    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368633    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:45:11.373585 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368635    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368638    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368641    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368643    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368646    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368649    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368651    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368654    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368656    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368658    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368661    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368663    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368666    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368669    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368671    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368673    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368676    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368679    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368683    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:45:11.374049 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368686    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368689    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368692    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368695    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368697    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368700    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368703    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:11.368705    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.368710    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.368823    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.370693    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.371855    2578 server.go:1019] "Starting client certificate rotation"
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.371954    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 20:45:11.374517 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.371989    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 20:45:11.374842 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.373665    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 20:45:11.374842 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.374780    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 20:45:11.379419 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.379404    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 28 20:45:11.382382 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.382365    2578 log.go:25] "Validated CRI v1 image API"
Jul 28 20:45:11.383150 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.383136    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 20:45:11.385237 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.385220    2578 fs.go:135] Filesystem UUIDs: map[60a82f27-0a7e-45aa-ad20-d0d8d0188d94:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f1ec148e-0390-4a06-9bcf-2ddf088e9166:/dev/nvme0n1p4]
Jul 28 20:45:11.385281 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.385237    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 20:45:11.390187 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.390060    2578 manager.go:216] Machine: {Timestamp:2026-07-28 20:45:11.388983618 +0000 UTC m=+0.243827521 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3135718 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21ce8f34e3e25ad72e054d4d8e82f0 SystemUUID:ec21ce8f-34e3-e25a-d72e-054d4d8e82f0 BootID:bf786d97-8aa0-47db-acac-bc4fa7a66639 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:46:bb:26:6f:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:46:bb:26:6f:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:e5:34:ce:80:b7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 20:45:11.390187 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.390183    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 20:45:11.390300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.390254    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 20:45:11.392696 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.392675    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 20:45:11.392818 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.392699    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-164.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 20:45:11.393699 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.393689    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 20:45:11.393731 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.393701    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 20:45:11.393731 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.393708    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 20:45:11.394236 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.394227    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 20:45:11.395260 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.395251    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 20:45:11.398771 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.398758    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 20:45:11.398921 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.398911    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 20:45:11.399430 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.399420    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 20:45:11.399462 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.399435    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 20:45:11.399462 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.399446    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 20:45:11.399462 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.399455    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 28 20:45:11.399573 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.399465    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 20:45:11.399809 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.399798    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 20:45:11.399871 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.399815    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 20:45:11.401477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.401463    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 20:45:11.402712 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.402697    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 20:45:11.402769 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.402738    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 20:45:11.403290 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403277    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 20:45:11.403329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403296    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 20:45:11.403329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403301    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 20:45:11.403329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403307    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 20:45:11.403329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403319    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 20:45:11.403329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403328    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 20:45:11.403468 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403339    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 20:45:11.403468 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403344    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 20:45:11.403468 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403350    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 20:45:11.403468 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403356    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 20:45:11.403468 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403365    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 20:45:11.403468 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403370    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 20:45:11.403908 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403899    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 20:45:11.403952 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.403910    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 20:45:11.405295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.405275    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 20:45:11.407548 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.407534    2578 server.go:1268] "Started kubelet"
Jul 28 20:45:11.407625 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.407596    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 20:45:11.407674 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.407632    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 20:45:11.407725 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.407696    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 20:45:11.408377 ip-10-0-138-164 systemd[1]: Started Kubernetes Kubelet.
Jul 28 20:45:11.412944 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.412922    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 20:45:11.413286 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.413265    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 20:45:11.414043 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.414018    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 20:45:11.414674 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.414659    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-164.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 20:45:11.414741 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.414712    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-164.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 20:45:11.414964 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.414948    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 20:45:11.415342 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.414665    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-164.ec2.internal.18c68efb1911b878  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-164.ec2.internal,UID:ip-10-0-138-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-164.ec2.internal,},FirstTimestamp:2026-07-28 20:45:11.407507576 +0000 UTC m=+0.262351479,LastTimestamp:2026-07-28 20:45:11.407507576 +0000 UTC m=+0.262351479,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-164.ec2.internal,}"
Jul 28 20:45:11.418202 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.418182    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 20:45:11.418202 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.418195    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 20:45:11.418354 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.418322    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:11.418715 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.418693    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 20:45:11.418715 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.418694    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 20:45:11.418837 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.418726    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 20:45:11.418882 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.418842    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 20:45:11.418882 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.418850    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 20:45:11.419033 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.419010    2578 factory.go:55] Registering systemd factory
Jul 28 20:45:11.419097 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.419052    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 28 20:45:11.419337 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.419321    2578 factory.go:153] Registering CRI-O factory
Jul 28 20:45:11.419406 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.419341    2578 factory.go:223] Registration of the crio container factory successfully
Jul 28 20:45:11.419406 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.419392    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 20:45:11.419471 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.419417    2578 factory.go:103] Registering Raw factory
Jul 28 20:45:11.419471 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.419429    2578 manager.go:1196] Started watching for new ooms in manager
Jul 28 20:45:11.419860 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.419835    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 20:45:11.419935 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.419838    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-164.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 20:45:11.420061 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.420048    2578 manager.go:349] Starting recovery of all containers
Jul 28 20:45:11.427363 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.427346    2578 manager.go:354] Recovery completed
Jul 28 20:45:11.429712 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.429557    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rd4rn"
Jul 28 20:45:11.432659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.432649    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:45:11.434871 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.434853    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:45:11.434944 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.434886    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:45:11.434944 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.434897    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:45:11.435452 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.435439    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 20:45:11.435452 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.435452    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 20:45:11.435543 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.435484    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 20:45:11.435543 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.435444    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:45:11.436166 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.436153    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:45:11.436226 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.436177    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:45:11.436226 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.436187    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:45:11.437665 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.437652    2578 policy_none.go:49] "None policy: Start"
Jul 28 20:45:11.437705 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.437670    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 20:45:11.437705 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.437681    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 20:45:11.439524 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.439507    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rd4rn"
Jul 28 20:45:11.440269 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.440255    2578 policy_none.go:47] "Start"
Jul 28 20:45:11.444062 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.443968    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-164.ec2.internal.18c68efb1ab3426a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-164.ec2.internal,UID:ip-10-0-138-164.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-164.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-164.ec2.internal,},FirstTimestamp:2026-07-28 20:45:11.434871402 +0000 UTC m=+0.289715305,LastTimestamp:2026-07-28 20:45:11.434871402 +0000 UTC m=+0.289715305,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-164.ec2.internal,}"
Jul 28 20:45:11.476616 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.476601    2578 manager.go:337] "Starting Device Plugin manager"
Jul 28 20:45:11.476693 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.476632    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 20:45:11.476693 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.476642    2578 server.go:87] "Starting device plugin registration server"
Jul 28 20:45:11.476908 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.476895    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 20:45:11.476972 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.476910    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 20:45:11.477018 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.476994    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 20:45:11.477126 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.477112    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 20:45:11.477196 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.477129    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 20:45:11.477558 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.477536    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 20:45:11.477621 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.477587    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:11.521600 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.521571    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 20:45:11.528759 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.522870    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 20:45:11.528759 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.522906    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 20:45:11.528759 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.522929    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 20:45:11.528759 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.522961    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 20:45:11.528759 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.525920    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:45:11.577686 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.577664    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:45:11.578467 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.578437    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:45:11.578467 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.578469    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:45:11.578617 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.578480    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:45:11.578617 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.578504    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.588418 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.588397    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.588502 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.588423    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-164.ec2.internal\": node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:11.613846 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.613789    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:11.623021 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.622990    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal"]
Jul 28 20:45:11.623091 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.623069    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:45:11.624426 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.624409    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:45:11.624479 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.624437    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:45:11.624479 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.624446    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:45:11.626735 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.626723    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:45:11.626865 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.626850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.626925 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.626887    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:45:11.627419 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.627391    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:45:11.627537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.627437    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:45:11.627537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.627451    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:45:11.627537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.627405    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:45:11.627537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.627513    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:45:11.627537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.627523    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:45:11.629837 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.629815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.629888 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.629847    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:45:11.630622 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.630607    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:45:11.630670 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.630634    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:45:11.630670 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.630645    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:45:11.646601 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.646584    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-164.ec2.internal\" not found" node="ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.650304 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.650290    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-164.ec2.internal\" not found" node="ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.714191 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.714168    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:11.720451 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.720429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18a744e7b35568a34a35b17c19e9750c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-164.ec2.internal\" (UID: \"18a744e7b35568a34a35b17c19e9750c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.720504 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.720453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c29c2ef69ffb500d6d735a526717d1b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal\" (UID: \"c29c2ef69ffb500d6d735a526717d1b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.720504 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.720471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29c2ef69ffb500d6d735a526717d1b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal\" (UID: \"c29c2ef69ffb500d6d735a526717d1b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.814533 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.814500    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:11.820824 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.820810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18a744e7b35568a34a35b17c19e9750c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-164.ec2.internal\" (UID: \"18a744e7b35568a34a35b17c19e9750c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.820885 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.820831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c29c2ef69ffb500d6d735a526717d1b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal\" (UID: \"c29c2ef69ffb500d6d735a526717d1b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.820885 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.820847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29c2ef69ffb500d6d735a526717d1b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal\" (UID: \"c29c2ef69ffb500d6d735a526717d1b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.820962 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.820886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29c2ef69ffb500d6d735a526717d1b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal\" (UID: \"c29c2ef69ffb500d6d735a526717d1b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.820962 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.820914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18a744e7b35568a34a35b17c19e9750c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-164.ec2.internal\" (UID: \"18a744e7b35568a34a35b17c19e9750c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.820962 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.820916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c29c2ef69ffb500d6d735a526717d1b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal\" (UID: \"c29c2ef69ffb500d6d735a526717d1b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.915248 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:11.915175    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:11.948636 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.948617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:11.954434 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:11.953791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:12.015881 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:12.015839    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:12.116350 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:12.116317    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:12.216848 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:12.216777    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:12.275289 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.275264    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:45:12.317184 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:12.317154    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:12.371701 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.371673    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 20:45:12.372146 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.371808    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 20:45:12.372146 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.371842    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 20:45:12.417741 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:12.417715    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-164.ec2.internal\" not found"
Jul 28 20:45:12.443831 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.443801    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 20:40:12 +0000 UTC" deadline="2028-01-23 21:59:59.719344495 +0000 UTC"
Jul 28 20:45:12.443831 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.443827    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13057h14m47.275521186s"
Jul 28 20:45:12.459420 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.459403    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:45:12.474998 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.474977    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:45:12.518574 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.518550    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:12.526716 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.526700    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 20:45:12.527429 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.527417    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal"
Jul 28 20:45:12.539142 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.539126    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 20:45:12.555208 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:12.555185    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a744e7b35568a34a35b17c19e9750c.slice/crio-47b8474ffffe6f3b973d4de226db1214f36f1beb3e1a573b0b6b131b88ee0665 WatchSource:0}: Error finding container 47b8474ffffe6f3b973d4de226db1214f36f1beb3e1a573b0b6b131b88ee0665: Status 404 returned error can't find the container with id 47b8474ffffe6f3b973d4de226db1214f36f1beb3e1a573b0b6b131b88ee0665
Jul 28 20:45:12.558897 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:12.558879    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:45:13.402264 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.401861    2578 apiserver.go:52] "Watching apiserver"
Jul 28 20:45:13.412684 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.412660    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 20:45:13.413721 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.413683    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-x5m6c","kube-system/global-pull-secret-syncer-v882q","kube-system/konnectivity-agent-lx58r","kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5","openshift-cluster-node-tuning-operator/tuned-vhls6","openshift-multus/multus-kkgdz","openshift-network-diagnostics/network-check-target-tsl56","openshift-network-operator/iptables-alerter-ftxmm","openshift-dns/node-resolver-r9wb9","openshift-image-registry/node-ca-8hrfs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal","openshift-multus/multus-additional-cni-plugins-94vbq","openshift-multus/network-metrics-daemon-9j2ft"]
Jul 28 20:45:13.416826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.416801    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.418948 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.418926    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 20:45:13.419071 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.418973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.419999 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.419860    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gksv2\""
Jul 28 20:45:13.419999 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.419883    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 20:45:13.419999 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.419890    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 20:45:13.421306 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.421284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:13.424147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.422179    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 20:45:13.424147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.422260    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 20:45:13.424147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.422437    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 20:45:13.424147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.422555    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 20:45:13.424147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.423010    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 20:45:13.424147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.423021    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mzh6r\""
Jul 28 20:45:13.424147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.423631    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 20:45:13.424147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.423856    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 20:45:13.424147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.424006    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7zrfv\""
Jul 28 20:45:13.424147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.424177    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:13.424620 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.424278    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:13.426178 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.426155    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ec92d4e-94e5-4437-96e2-22a783dc73a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d014f55f-c43a-4ef5-bd1e-e517a72ea8a1-agent-certs\") pod \"konnectivity-agent-lx58r\" (UID: \"d014f55f-c43a-4ef5-bd1e-e517a72ea8a1\") " pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhnm\" (UniqueName: \"kubernetes.io/projected/87a4da6a-0543-4046-b35d-00579314cafe-kube-api-access-hxhnm\") pod \"node-resolver-r9wb9\" (UID: \"87a4da6a-0543-4046-b35d-00579314cafe\") " pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-cnibin\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7wh\" (UniqueName: \"kubernetes.io/projected/1ec92d4e-94e5-4437-96e2-22a783dc73a6-kube-api-access-fp7wh\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ec92d4e-94e5-4437-96e2-22a783dc73a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a4da6a-0543-4046-b35d-00579314cafe-tmp-dir\") pod \"node-resolver-r9wb9\" (UID: \"87a4da6a-0543-4046-b35d-00579314cafe\") " pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-system-cni-dir\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ec92d4e-94e5-4437-96e2-22a783dc73a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d014f55f-c43a-4ef5-bd1e-e517a72ea8a1-konnectivity-ca\") pod \"konnectivity-agent-lx58r\" (UID: \"d014f55f-c43a-4ef5-bd1e-e517a72ea8a1\") " pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87a4da6a-0543-4046-b35d-00579314cafe-hosts-file\") pod \"node-resolver-r9wb9\" (UID: \"87a4da6a-0543-4046-b35d-00579314cafe\") " pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.427980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.427943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-os-release\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.428805 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.428787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.429562 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.429395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 20:45:13.429562 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.429439    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 20:45:13.429562 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.429476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 20:45:13.429756 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.429695    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cmx86\""
Jul 28 20:45:13.433268 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.430831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.433268 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.431460    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n9vpq\""
Jul 28 20:45:13.433268 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.431467    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:45:13.433268 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.431575    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 20:45:13.433268 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.433184    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:45:13.433560 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.433485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.433893 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.433873    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 20:45:13.434204 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.434181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 20:45:13.434204 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.434184    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7m9zj\""
Jul 28 20:45:13.435401 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.435219    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 20:45:13.439598 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.438505    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 20:45:13.439835 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.439814    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gn8pp\""
Jul 28 20:45:13.440263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.440245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:13.440507 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.440480    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:13.443495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.443475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.445863 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.445828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.447482 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.447461    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 20:45:13.447677 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.447660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 20:45:13.447879 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.447858    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cwsxq\""
Jul 28 20:45:13.448131 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.448111    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 20:45:13.448431 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.448413    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 20:45:13.448520 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.448487    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 20:45:13.448665 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.448646    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 20:45:13.448744 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.448728    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 20:45:13.448888 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.448873    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4kx8w\""
Jul 28 20:45:13.449027 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.448653    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 20:45:13.449064 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.449014    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 20:45:13.450493 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.450472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:13.450812 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.450563    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:13.462193 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.462170    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9zsgl"
Jul 28 20:45:13.472388 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.472018    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9zsgl"
Jul 28 20:45:13.520573 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.520546    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 20:45:13.528161 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-sys\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.528312 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-lib-modules\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.528312 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal" event={"ID":"18a744e7b35568a34a35b17c19e9750c","Type":"ContainerStarted","Data":"47b8474ffffe6f3b973d4de226db1214f36f1beb3e1a573b0b6b131b88ee0665"}
Jul 28 20:45:13.528312 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-os-release\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.528312 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpkwj\" (UniqueName: \"kubernetes.io/projected/a068daf5-405c-4591-8ec2-cc5cb3bd545f-kube-api-access-tpkwj\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.528312 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de6d5b34-f052-4613-a09c-4892d9fac92d-host\") pod \"node-ca-8hrfs\" (UID: \"de6d5b34-f052-4613-a09c-4892d9fac92d\") " pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.528597 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-kubernetes\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.528597 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-host\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.528597 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7sw6\" (UniqueName: \"kubernetes.io/projected/ea587c17-0489-450d-a08b-99232d028831-kube-api-access-h7sw6\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.528597 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-multus-socket-dir-parent\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.528597 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-etc-kubernetes\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.528597 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-run-netns\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.528597 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4096284-810e-4423-a040-0658e955aceb-multus-daemon-config\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.528954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-run-multus-certs\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.528954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-etc-openvswitch\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.528954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-log-socket\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.528954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87a4da6a-0543-4046-b35d-00579314cafe-hosts-file\") pod \"node-resolver-r9wb9\" (UID: \"87a4da6a-0543-4046-b35d-00579314cafe\") " pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.528954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-os-release\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.528954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528828    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87a4da6a-0543-4046-b35d-00579314cafe-hosts-file\") pod \"node-resolver-r9wb9\" (UID: \"87a4da6a-0543-4046-b35d-00579314cafe\") " pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.528954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-sysctl-conf\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.528954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-var-lib-kubelet\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.528954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-system-cni-dir\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.528969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-var-lib-kubelet\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-cni-bin\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-os-release\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-sysctl-d\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-kubelet\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-multus-conf-dir\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rftbj\" (UniqueName: \"kubernetes.io/projected/81776a2b-e432-46b1-beec-f76495c84ad2-kube-api-access-rftbj\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-multus-cni-dir\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-var-lib-openvswitch\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.529436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a068daf5-405c-4591-8ec2-cc5cb3bd545f-ovn-node-metrics-cert\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-run\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-run-k8s-cni-cncf-io\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-slash\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-node-log\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw2jz\" (UniqueName: \"kubernetes.io/projected/de6d5b34-f052-4613-a09c-4892d9fac92d-kube-api-access-lw2jz\") pod \"node-ca-8hrfs\" (UID: \"de6d5b34-f052-4613-a09c-4892d9fac92d\") " pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ec92d4e-94e5-4437-96e2-22a783dc73a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d014f55f-c43a-4ef5-bd1e-e517a72ea8a1-agent-certs\") pod \"konnectivity-agent-lx58r\" (UID: \"d014f55f-c43a-4ef5-bd1e-e517a72ea8a1\") " pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-systemd\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-hostroot\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gd59\" (UniqueName: \"kubernetes.io/projected/b4096284-810e-4423-a040-0658e955aceb-kube-api-access-2gd59\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-run-openvswitch\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-run-ovn\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7wh\" (UniqueName: \"kubernetes.io/projected/1ec92d4e-94e5-4437-96e2-22a783dc73a6-kube-api-access-fp7wh\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.529941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.529929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf7gc\" (UniqueName: \"kubernetes.io/projected/446e31b8-b81d-461d-8f23-acb33808b1e9-kube-api-access-wf7gc\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4096284-810e-4423-a040-0658e955aceb-cni-binary-copy\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf9fedae-ce04-423f-aefb-979d1cd5ad5a-host-slash\") pod \"iptables-alerter-ftxmm\" (UID: \"cf9fedae-ce04-423f-aefb-979d1cd5ad5a\") " pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-systemd-units\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a068daf5-405c-4591-8ec2-cc5cb3bd545f-env-overrides\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-system-cni-dir\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ec92d4e-94e5-4437-96e2-22a783dc73a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d014f55f-c43a-4ef5-bd1e-e517a72ea8a1-konnectivity-ca\") pod \"konnectivity-agent-lx58r\" (UID: \"d014f55f-c43a-4ef5-bd1e-e517a72ea8a1\") " pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530255    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-cni-netd\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ec92d4e-94e5-4437-96e2-22a783dc73a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-var-lib-cni-multus\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-run-netns\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-device-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-sys-fs\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-sysconfig\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea587c17-0489-450d-a08b-99232d028831-etc-tuned\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.530726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea587c17-0489-450d-a08b-99232d028831-tmp\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.531477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e665f43b-b228-44a5-a9ff-412e77074d0d-kubelet-config\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:13.531477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhnm\" (UniqueName: \"kubernetes.io/projected/87a4da6a-0543-4046-b35d-00579314cafe-kube-api-access-hxhnm\") pod \"node-resolver-r9wb9\" (UID: \"87a4da6a-0543-4046-b35d-00579314cafe\") " pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.531477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.530870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-system-cni-dir\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.531477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.531078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-modprobe-d\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.531477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.531117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-run-systemd\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.531477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.531191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a068daf5-405c-4591-8ec2-cc5cb3bd545f-ovnkube-script-lib\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.531477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.531261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lds\" (UniqueName: \"kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds\") pod \"network-check-target-tsl56\" (UID: \"9192a864-1460-4306-b8b8-c2b51fabe108\") " pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:13.531477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.531362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d014f55f-c43a-4ef5-bd1e-e517a72ea8a1-konnectivity-ca\") pod \"konnectivity-agent-lx58r\" (UID: \"d014f55f-c43a-4ef5-bd1e-e517a72ea8a1\") " pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:13.531887 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.531819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-cnibin\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.531887 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.531878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-registration-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.531993 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.531905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.531993 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.531950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ec92d4e-94e5-4437-96e2-22a783dc73a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.531993 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.531979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a4da6a-0543-4046-b35d-00579314cafe-tmp-dir\") pod \"node-resolver-r9wb9\" (UID: \"87a4da6a-0543-4046-b35d-00579314cafe\") " pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.532164 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-socket-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.532606 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf9fedae-ce04-423f-aefb-979d1cd5ad5a-iptables-alerter-script\") pod \"iptables-alerter-ftxmm\" (UID: \"cf9fedae-ce04-423f-aefb-979d1cd5ad5a\") " pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.532698 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:13.532698 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sngd\" (UniqueName: \"kubernetes.io/projected/cf9fedae-ce04-423f-aefb-979d1cd5ad5a-kube-api-access-9sngd\") pod \"iptables-alerter-ftxmm\" (UID: \"cf9fedae-ce04-423f-aefb-979d1cd5ad5a\") " pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.532698 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e665f43b-b228-44a5-a9ff-412e77074d0d-dbus\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:13.532805 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.532805 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a068daf5-405c-4591-8ec2-cc5cb3bd545f-ovnkube-config\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.532805 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de6d5b34-f052-4613-a09c-4892d9fac92d-serviceca\") pod \"node-ca-8hrfs\" (UID: \"de6d5b34-f052-4613-a09c-4892d9fac92d\") " pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.532903 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-var-lib-cni-bin\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.532953 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-cnibin\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.533897 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.533624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ec92d4e-94e5-4437-96e2-22a783dc73a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.533897 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.533850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a4da6a-0543-4046-b35d-00579314cafe-tmp-dir\") pod \"node-resolver-r9wb9\" (UID: \"87a4da6a-0543-4046-b35d-00579314cafe\") " pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.534069 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.532888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-cnibin\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.534145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.534082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.534197 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.534177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.536425 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.534413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ec92d4e-94e5-4437-96e2-22a783dc73a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.537019 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.536989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ec92d4e-94e5-4437-96e2-22a783dc73a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.538705 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.538680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d014f55f-c43a-4ef5-bd1e-e517a72ea8a1-agent-certs\") pod \"konnectivity-agent-lx58r\" (UID: \"d014f55f-c43a-4ef5-bd1e-e517a72ea8a1\") " pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:13.543201 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.543178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7wh\" (UniqueName: \"kubernetes.io/projected/1ec92d4e-94e5-4437-96e2-22a783dc73a6-kube-api-access-fp7wh\") pod \"multus-additional-cni-plugins-94vbq\" (UID: \"1ec92d4e-94e5-4437-96e2-22a783dc73a6\") " pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.544307 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.544283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhnm\" (UniqueName: \"kubernetes.io/projected/87a4da6a-0543-4046-b35d-00579314cafe-kube-api-access-hxhnm\") pod \"node-resolver-r9wb9\" (UID: \"87a4da6a-0543-4046-b35d-00579314cafe\") " pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.550504 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:13.550479    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29c2ef69ffb500d6d735a526717d1b8.slice/crio-9781cb5dfaee8d04c550e9a8a707de271e0e465554d22be77b3c50438c604317 WatchSource:0}: Error finding container 9781cb5dfaee8d04c550e9a8a707de271e0e465554d22be77b3c50438c604317: Status 404 returned error can't find the container with id 9781cb5dfaee8d04c550e9a8a707de271e0e465554d22be77b3c50438c604317
Jul 28 20:45:13.634857 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.634817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-systemd-units\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.634857 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.634863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a068daf5-405c-4591-8ec2-cc5cb3bd545f-env-overrides\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.635075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.634926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-systemd-units\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.635075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.634994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-cni-netd\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.635075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-var-lib-cni-multus\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.635075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-run-netns\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-cni-netd\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-device-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-sys-fs\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-var-lib-cni-multus\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-sysconfig\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-run-netns\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-sysconfig\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea587c17-0489-450d-a08b-99232d028831-etc-tuned\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea587c17-0489-450d-a08b-99232d028831-tmp\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-device-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.635295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e665f43b-b228-44a5-a9ff-412e77074d0d-kubelet-config\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-modprobe-d\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635319    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-sys-fs\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-run-systemd\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a068daf5-405c-4591-8ec2-cc5cb3bd545f-ovnkube-script-lib\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lds\" (UniqueName: \"kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds\") pod \"network-check-target-tsl56\" (UID: \"9192a864-1460-4306-b8b8-c2b51fabe108\") " pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a068daf5-405c-4591-8ec2-cc5cb3bd545f-env-overrides\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-cnibin\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-registration-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-registration-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-modprobe-d\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-socket-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-run-systemd\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf9fedae-ce04-423f-aefb-979d1cd5ad5a-iptables-alerter-script\") pod \"iptables-alerter-ftxmm\" (UID: \"cf9fedae-ce04-423f-aefb-979d1cd5ad5a\") " pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sngd\" (UniqueName: \"kubernetes.io/projected/cf9fedae-ce04-423f-aefb-979d1cd5ad5a-kube-api-access-9sngd\") pod \"iptables-alerter-ftxmm\" (UID: \"cf9fedae-ce04-423f-aefb-979d1cd5ad5a\") " pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.635826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e665f43b-b228-44a5-a9ff-412e77074d0d-kubelet-config\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e665f43b-b228-44a5-a9ff-412e77074d0d-dbus\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a068daf5-405c-4591-8ec2-cc5cb3bd545f-ovnkube-config\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-cnibin\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de6d5b34-f052-4613-a09c-4892d9fac92d-serviceca\") pod \"node-ca-8hrfs\" (UID: \"de6d5b34-f052-4613-a09c-4892d9fac92d\") " pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-socket-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-var-lib-cni-bin\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.635884    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-sys\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-lib-modules\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-os-release\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpkwj\" (UniqueName: \"kubernetes.io/projected/a068daf5-405c-4591-8ec2-cc5cb3bd545f-kube-api-access-tpkwj\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.636123    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs podName:81776a2b-e432-46b1-beec-f76495c84ad2 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:14.135997575 +0000 UTC m=+2.990841468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs") pod "network-metrics-daemon-9j2ft" (UID: "81776a2b-e432-46b1-beec-f76495c84ad2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de6d5b34-f052-4613-a09c-4892d9fac92d-host\") pod \"node-ca-8hrfs\" (UID: \"de6d5b34-f052-4613-a09c-4892d9fac92d\") " pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de6d5b34-f052-4613-a09c-4892d9fac92d-serviceca\") pod \"node-ca-8hrfs\" (UID: \"de6d5b34-f052-4613-a09c-4892d9fac92d\") " pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.636695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-kubernetes\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.635665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-host\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7sw6\" (UniqueName: \"kubernetes.io/projected/ea587c17-0489-450d-a08b-99232d028831-kube-api-access-h7sw6\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-multus-socket-dir-parent\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-etc-kubernetes\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-run-netns\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4096284-810e-4423-a040-0658e955aceb-multus-daemon-config\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-run-multus-certs\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-etc-openvswitch\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-log-socket\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-sysctl-conf\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-var-lib-kubelet\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a068daf5-405c-4591-8ec2-cc5cb3bd545f-ovnkube-script-lib\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-sys\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-system-cni-dir\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-var-lib-kubelet\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.637619 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-system-cni-dir\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-cni-bin\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de6d5b34-f052-4613-a09c-4892d9fac92d-host\") pod \"node-ca-8hrfs\" (UID: \"de6d5b34-f052-4613-a09c-4892d9fac92d\") " pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-cni-bin\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-sysctl-d\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf9fedae-ce04-423f-aefb-979d1cd5ad5a-iptables-alerter-script\") pod \"iptables-alerter-ftxmm\" (UID: \"cf9fedae-ce04-423f-aefb-979d1cd5ad5a\") " pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-var-lib-cni-bin\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a068daf5-405c-4591-8ec2-cc5cb3bd545f-ovnkube-config\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636854    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-var-lib-kubelet\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-kubelet\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.636960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-os-release\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/446e31b8-b81d-461d-8f23-acb33808b1e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e665f43b-b228-44a5-a9ff-412e77074d0d-dbus\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637039    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-run-multus-certs\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-log-socket\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-multus-socket-dir-parent\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-kubelet\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.638497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-kubernetes\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-run-netns\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-etc-openvswitch\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-multus-conf-dir\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-host\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rftbj\" (UniqueName: \"kubernetes.io/projected/81776a2b-e432-46b1-beec-f76495c84ad2-kube-api-access-rftbj\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-var-lib-kubelet\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-multus-cni-dir\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-var-lib-openvswitch\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-sysctl-d\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-multus-conf-dir\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a068daf5-405c-4591-8ec2-cc5cb3bd545f-ovn-node-metrics-cert\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-etc-kubernetes\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-run\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-var-lib-openvswitch\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-run-k8s-cni-cncf-io\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-host-run-k8s-cni-cncf-io\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-slash\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.639335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-multus-cni-dir\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-node-log\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4096284-810e-4423-a040-0658e955aceb-multus-daemon-config\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw2jz\" (UniqueName: \"kubernetes.io/projected/de6d5b34-f052-4613-a09c-4892d9fac92d-kube-api-access-lw2jz\") pod \"node-ca-8hrfs\" (UID: \"de6d5b34-f052-4613-a09c-4892d9fac92d\") " pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-systemd\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.637664    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-hostroot\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-lib-modules\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.637717    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret podName:e665f43b-b228-44a5-a9ff-412e77074d0d nodeName:}" failed. No retries permitted until 2026-07-28 20:45:14.137701247 +0000 UTC m=+2.992545138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret") pod "global-pull-secret-syncer-v882q" (UID: "e665f43b-b228-44a5-a9ff-412e77074d0d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gd59\" (UniqueName: \"kubernetes.io/projected/b4096284-810e-4423-a040-0658e955aceb-kube-api-access-2gd59\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4096284-810e-4423-a040-0658e955aceb-hostroot\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-run\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea587c17-0489-450d-a08b-99232d028831-etc-tuned\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-slash\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-run-openvswitch\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-node-log\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-run-ovn\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-systemd\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.640075 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7gc\" (UniqueName: \"kubernetes.io/projected/446e31b8-b81d-461d-8f23-acb33808b1e9-kube-api-access-wf7gc\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4096284-810e-4423-a040-0658e955aceb-cni-binary-copy\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.637973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea587c17-0489-450d-a08b-99232d028831-etc-sysctl-conf\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.638023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-run-ovn\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.638053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf9fedae-ce04-423f-aefb-979d1cd5ad5a-host-slash\") pod \"iptables-alerter-ftxmm\" (UID: \"cf9fedae-ce04-423f-aefb-979d1cd5ad5a\") " pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.638208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf9fedae-ce04-423f-aefb-979d1cd5ad5a-host-slash\") pod \"iptables-alerter-ftxmm\" (UID: \"cf9fedae-ce04-423f-aefb-979d1cd5ad5a\") " pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.638223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a068daf5-405c-4591-8ec2-cc5cb3bd545f-run-openvswitch\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.638449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4096284-810e-4423-a040-0658e955aceb-cni-binary-copy\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.639970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea587c17-0489-450d-a08b-99232d028831-tmp\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.640827 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.639983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a068daf5-405c-4591-8ec2-cc5cb3bd545f-ovn-node-metrics-cert\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.652983 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.652912    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:45:13.652983 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.652945    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:45:13.652983 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.652962    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f4lds for pod openshift-network-diagnostics/network-check-target-tsl56: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:13.653251 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:13.653043    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds podName:9192a864-1460-4306-b8b8-c2b51fabe108 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:14.153024121 +0000 UTC m=+3.007868012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f4lds" (UniqueName: "kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds") pod "network-check-target-tsl56" (UID: "9192a864-1460-4306-b8b8-c2b51fabe108") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:13.653518 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.653493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sngd\" (UniqueName: \"kubernetes.io/projected/cf9fedae-ce04-423f-aefb-979d1cd5ad5a-kube-api-access-9sngd\") pod \"iptables-alerter-ftxmm\" (UID: \"cf9fedae-ce04-423f-aefb-979d1cd5ad5a\") " pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.654416 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.654372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7gc\" (UniqueName: \"kubernetes.io/projected/446e31b8-b81d-461d-8f23-acb33808b1e9-kube-api-access-wf7gc\") pod \"aws-ebs-csi-driver-node-9xzq5\" (UID: \"446e31b8-b81d-461d-8f23-acb33808b1e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.654521 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.654484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gd59\" (UniqueName: \"kubernetes.io/projected/b4096284-810e-4423-a040-0658e955aceb-kube-api-access-2gd59\") pod \"multus-kkgdz\" (UID: \"b4096284-810e-4423-a040-0658e955aceb\") " pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.655281 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.655258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw2jz\" (UniqueName: \"kubernetes.io/projected/de6d5b34-f052-4613-a09c-4892d9fac92d-kube-api-access-lw2jz\") pod \"node-ca-8hrfs\" (UID: \"de6d5b34-f052-4613-a09c-4892d9fac92d\") " pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.659621 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.659598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7sw6\" (UniqueName: \"kubernetes.io/projected/ea587c17-0489-450d-a08b-99232d028831-kube-api-access-h7sw6\") pod \"tuned-vhls6\" (UID: \"ea587c17-0489-450d-a08b-99232d028831\") " pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.660688 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.660654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rftbj\" (UniqueName: \"kubernetes.io/projected/81776a2b-e432-46b1-beec-f76495c84ad2-kube-api-access-rftbj\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:13.662130 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.662109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpkwj\" (UniqueName: \"kubernetes.io/projected/a068daf5-405c-4591-8ec2-cc5cb3bd545f-kube-api-access-tpkwj\") pod \"ovnkube-node-x5m6c\" (UID: \"a068daf5-405c-4591-8ec2-cc5cb3bd545f\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.732270 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.732234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9wb9"
Jul 28 20:45:13.742623 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.742591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94vbq"
Jul 28 20:45:13.754696 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.754672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:13.761488 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.761471    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5"
Jul 28 20:45:13.769282 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.769253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vhls6"
Jul 28 20:45:13.778930 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.778911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ftxmm"
Jul 28 20:45:13.787780 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.787759    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kkgdz"
Jul 28 20:45:13.795596 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.795568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:13.802578 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.802557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8hrfs"
Jul 28 20:45:13.903738 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:13.903656    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:45:14.142672 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.142637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:14.142839 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.142689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:14.142839 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:14.142819    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:14.142839 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:14.142826    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:14.142975 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:14.142885    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret podName:e665f43b-b228-44a5-a9ff-412e77074d0d nodeName:}" failed. No retries permitted until 2026-07-28 20:45:15.142868185 +0000 UTC m=+3.997712076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret") pod "global-pull-secret-syncer-v882q" (UID: "e665f43b-b228-44a5-a9ff-412e77074d0d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:14.142975 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:14.142900    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs podName:81776a2b-e432-46b1-beec-f76495c84ad2 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:15.142894208 +0000 UTC m=+3.997738098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs") pod "network-metrics-daemon-9j2ft" (UID: "81776a2b-e432-46b1-beec-f76495c84ad2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:14.243860 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.243816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lds\" (UniqueName: \"kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds\") pod \"network-check-target-tsl56\" (UID: \"9192a864-1460-4306-b8b8-c2b51fabe108\") " pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:14.244036 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:14.243990    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:45:14.244036 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:14.244018    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:45:14.244036 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:14.244029    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f4lds for pod openshift-network-diagnostics/network-check-target-tsl56: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:14.244214 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:14.244121    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds podName:9192a864-1460-4306-b8b8-c2b51fabe108 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:15.244081573 +0000 UTC m=+4.098925467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f4lds" (UniqueName: "kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds") pod "network-check-target-tsl56" (UID: "9192a864-1460-4306-b8b8-c2b51fabe108") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:14.407791 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:14.407755    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea587c17_0489_450d_a08b_99232d028831.slice/crio-ccfc29dc012bc3849c44c22f787fa067df40f4fcda56ac8d9d7234e7ec2a6ab1 WatchSource:0}: Error finding container ccfc29dc012bc3849c44c22f787fa067df40f4fcda56ac8d9d7234e7ec2a6ab1: Status 404 returned error can't find the container with id ccfc29dc012bc3849c44c22f787fa067df40f4fcda56ac8d9d7234e7ec2a6ab1
Jul 28 20:45:14.409281 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:14.409143    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6d5b34_f052_4613_a09c_4892d9fac92d.slice/crio-97ab2d18ad3dabc574042c1583857e2e89a9d406407cecac5877c2424cbfd16a WatchSource:0}: Error finding container 97ab2d18ad3dabc574042c1583857e2e89a9d406407cecac5877c2424cbfd16a: Status 404 returned error can't find the container with id 97ab2d18ad3dabc574042c1583857e2e89a9d406407cecac5877c2424cbfd16a
Jul 28 20:45:14.416801 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:14.416771    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec92d4e_94e5_4437_96e2_22a783dc73a6.slice/crio-1e9715643e5bed2f97aad6828dc746253673129a1a40c63bdd18240a4bcc9a80 WatchSource:0}: Error finding container 1e9715643e5bed2f97aad6828dc746253673129a1a40c63bdd18240a4bcc9a80: Status 404 returned error can't find the container with id 1e9715643e5bed2f97aad6828dc746253673129a1a40c63bdd18240a4bcc9a80
Jul 28 20:45:14.417558 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:14.417528    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd014f55f_c43a_4ef5_bd1e_e517a72ea8a1.slice/crio-fd2631615430fb3bedadcacf13852a3ac516162d25cd92720b62e5befaffc82d WatchSource:0}: Error finding container fd2631615430fb3bedadcacf13852a3ac516162d25cd92720b62e5befaffc82d: Status 404 returned error can't find the container with id fd2631615430fb3bedadcacf13852a3ac516162d25cd92720b62e5befaffc82d
Jul 28 20:45:14.440549 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:14.440521    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a4da6a_0543_4046_b35d_00579314cafe.slice/crio-c1abb2e168dcefc98d598483c259de95fbfb308583e80721e0c2bc8a36b9c01e WatchSource:0}: Error finding container c1abb2e168dcefc98d598483c259de95fbfb308583e80721e0c2bc8a36b9c01e: Status 404 returned error can't find the container with id c1abb2e168dcefc98d598483c259de95fbfb308583e80721e0c2bc8a36b9c01e
Jul 28 20:45:14.441608 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:14.441565    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446e31b8_b81d_461d_8f23_acb33808b1e9.slice/crio-2457aa8cf27763995860ddcad7d0b99ef6173f3092fec9dc8854f4485c4e45af WatchSource:0}: Error finding container 2457aa8cf27763995860ddcad7d0b99ef6173f3092fec9dc8854f4485c4e45af: Status 404 returned error can't find the container with id 2457aa8cf27763995860ddcad7d0b99ef6173f3092fec9dc8854f4485c4e45af
Jul 28 20:45:14.475682 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.475636    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 20:40:14 +0000 UTC" deadline="2028-01-25 19:48:03.796648389 +0000 UTC"
Jul 28 20:45:14.475682 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.475681    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13103h2m49.320970135s"
Jul 28 20:45:14.523948 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.523925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:14.524105 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:14.524036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:14.532246 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.532211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vhls6" event={"ID":"ea587c17-0489-450d-a08b-99232d028831","Type":"ContainerStarted","Data":"ccfc29dc012bc3849c44c22f787fa067df40f4fcda56ac8d9d7234e7ec2a6ab1"}
Jul 28 20:45:14.533568 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.533536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal" event={"ID":"c29c2ef69ffb500d6d735a526717d1b8","Type":"ContainerStarted","Data":"9781cb5dfaee8d04c550e9a8a707de271e0e465554d22be77b3c50438c604317"}
Jul 28 20:45:14.534843 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.534799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kkgdz" event={"ID":"b4096284-810e-4423-a040-0658e955aceb","Type":"ContainerStarted","Data":"e9364cd6090a128ca27488024dbd04c186502aad912e58639c8a97e4f80bb856"}
Jul 28 20:45:14.536288 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.536261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c" event={"ID":"a068daf5-405c-4591-8ec2-cc5cb3bd545f","Type":"ContainerStarted","Data":"a1dff460a31c84dd9a6e2ef339c84a591e0bc156cbd1448f6544999ce0996d15"}
Jul 28 20:45:14.537701 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.537680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9wb9" event={"ID":"87a4da6a-0543-4046-b35d-00579314cafe","Type":"ContainerStarted","Data":"c1abb2e168dcefc98d598483c259de95fbfb308583e80721e0c2bc8a36b9c01e"}
Jul 28 20:45:14.538901 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.538879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lx58r" event={"ID":"d014f55f-c43a-4ef5-bd1e-e517a72ea8a1","Type":"ContainerStarted","Data":"fd2631615430fb3bedadcacf13852a3ac516162d25cd92720b62e5befaffc82d"}
Jul 28 20:45:14.540196 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.540168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ftxmm" event={"ID":"cf9fedae-ce04-423f-aefb-979d1cd5ad5a","Type":"ContainerStarted","Data":"80334413b468b350c6aa565a360f065fc3177e38430dae1370d6932d233354aa"}
Jul 28 20:45:14.541374 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.541341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8hrfs" event={"ID":"de6d5b34-f052-4613-a09c-4892d9fac92d","Type":"ContainerStarted","Data":"97ab2d18ad3dabc574042c1583857e2e89a9d406407cecac5877c2424cbfd16a"}
Jul 28 20:45:14.545007 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.544972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5" event={"ID":"446e31b8-b81d-461d-8f23-acb33808b1e9","Type":"ContainerStarted","Data":"2457aa8cf27763995860ddcad7d0b99ef6173f3092fec9dc8854f4485c4e45af"}
Jul 28 20:45:14.546123 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.546102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vbq" event={"ID":"1ec92d4e-94e5-4437-96e2-22a783dc73a6","Type":"ContainerStarted","Data":"1e9715643e5bed2f97aad6828dc746253673129a1a40c63bdd18240a4bcc9a80"}
Jul 28 20:45:14.565860 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:14.565812    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal" podStartSLOduration=1.5657961500000002 podStartE2EDuration="1.56579615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:14.565065832 +0000 UTC m=+3.419909749" watchObservedRunningTime="2026-07-28 20:45:14.56579615 +0000 UTC m=+3.420640133"
Jul 28 20:45:15.152212 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:15.150872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:15.152212 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:15.151274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:15.152212 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:15.151410    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:15.152212 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:15.151480    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret podName:e665f43b-b228-44a5-a9ff-412e77074d0d nodeName:}" failed. No retries permitted until 2026-07-28 20:45:17.151459988 +0000 UTC m=+6.006303982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret") pod "global-pull-secret-syncer-v882q" (UID: "e665f43b-b228-44a5-a9ff-412e77074d0d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:15.152212 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:15.151996    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:15.152212 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:15.152051    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs podName:81776a2b-e432-46b1-beec-f76495c84ad2 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:17.152032732 +0000 UTC m=+6.006876629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs") pod "network-metrics-daemon-9j2ft" (UID: "81776a2b-e432-46b1-beec-f76495c84ad2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:15.251700 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:15.251663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lds\" (UniqueName: \"kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds\") pod \"network-check-target-tsl56\" (UID: \"9192a864-1460-4306-b8b8-c2b51fabe108\") " pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:15.251891 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:15.251871    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:45:15.251973 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:15.251901    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:45:15.251973 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:15.251917    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f4lds for pod openshift-network-diagnostics/network-check-target-tsl56: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:15.252083 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:15.251989    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds podName:9192a864-1460-4306-b8b8-c2b51fabe108 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:17.251968439 +0000 UTC m=+6.106812333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f4lds" (UniqueName: "kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds") pod "network-check-target-tsl56" (UID: "9192a864-1460-4306-b8b8-c2b51fabe108") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:15.341595 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:15.341345    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:45:15.476213 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:15.476120    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 20:40:14 +0000 UTC" deadline="2028-03-04 09:57:11.946350637 +0000 UTC"
Jul 28 20:45:15.476213 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:15.476179    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14029h11m56.470177377s"
Jul 28 20:45:15.525836 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:15.524268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:15.525836 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:15.524420    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:15.525836 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:15.524473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:15.525836 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:15.524604    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:15.554266 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:15.554215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-164.ec2.internal" event={"ID":"18a744e7b35568a34a35b17c19e9750c","Type":"ContainerStarted","Data":"574151b9453a64de57b315998fcc8421084ac0d4dba7f906f9f77ea2628fce4b"}
Jul 28 20:45:16.526887 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:16.526856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:16.527354 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:16.526989    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:17.168487 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:17.168451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:17.168682 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:17.168520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:17.168682 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:17.168658    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:17.168807 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:17.168719    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret podName:e665f43b-b228-44a5-a9ff-412e77074d0d nodeName:}" failed. No retries permitted until 2026-07-28 20:45:21.16869787 +0000 UTC m=+10.023541776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret") pod "global-pull-secret-syncer-v882q" (UID: "e665f43b-b228-44a5-a9ff-412e77074d0d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:17.169243 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:17.169222    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:17.169355 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:17.169292    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs podName:81776a2b-e432-46b1-beec-f76495c84ad2 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:21.169274984 +0000 UTC m=+10.024118881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs") pod "network-metrics-daemon-9j2ft" (UID: "81776a2b-e432-46b1-beec-f76495c84ad2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:17.269041 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:17.269003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lds\" (UniqueName: \"kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds\") pod \"network-check-target-tsl56\" (UID: \"9192a864-1460-4306-b8b8-c2b51fabe108\") " pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:17.269257 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:17.269205    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:45:17.269257 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:17.269230    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:45:17.269257 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:17.269246    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f4lds for pod openshift-network-diagnostics/network-check-target-tsl56: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:17.269429 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:17.269309    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds podName:9192a864-1460-4306-b8b8-c2b51fabe108 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:21.269287147 +0000 UTC m=+10.124131125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f4lds" (UniqueName: "kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds") pod "network-check-target-tsl56" (UID: "9192a864-1460-4306-b8b8-c2b51fabe108") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:17.524266 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:17.524223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:17.524455 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:17.524385    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:17.524455 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:17.524433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:17.524597 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:17.524541    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:17.589280 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:17.589237    2578 generic.go:356] "Generic (PLEG): container finished" podID="c29c2ef69ffb500d6d735a526717d1b8" containerID="836a01ad1223213688cee807e3f8135aa5c2416cd522fd649bdafb3015d0fd94" exitCode=0
Jul 28 20:45:17.589826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:17.589301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal" event={"ID":"c29c2ef69ffb500d6d735a526717d1b8","Type":"ContainerDied","Data":"836a01ad1223213688cee807e3f8135aa5c2416cd522fd649bdafb3015d0fd94"}
Jul 28 20:45:18.523573 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:18.523480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:18.523777 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:18.523616    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:19.526184 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:19.525539    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:19.526184 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:19.525662    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:19.526184 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:19.526027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:19.526184 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:19.526135    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:20.523972 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:20.523696    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:20.523972 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:20.523848    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:21.202659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:21.202569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:21.203177 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:21.202676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:21.203177 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:21.202771    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:21.203177 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:21.202807    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:21.203177 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:21.202856    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs podName:81776a2b-e432-46b1-beec-f76495c84ad2 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:29.202834691 +0000 UTC m=+18.057678603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs") pod "network-metrics-daemon-9j2ft" (UID: "81776a2b-e432-46b1-beec-f76495c84ad2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:21.203177 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:21.202878    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret podName:e665f43b-b228-44a5-a9ff-412e77074d0d nodeName:}" failed. No retries permitted until 2026-07-28 20:45:29.202868052 +0000 UTC m=+18.057711942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret") pod "global-pull-secret-syncer-v882q" (UID: "e665f43b-b228-44a5-a9ff-412e77074d0d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:21.303687 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:21.303639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lds\" (UniqueName: \"kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds\") pod \"network-check-target-tsl56\" (UID: \"9192a864-1460-4306-b8b8-c2b51fabe108\") " pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:21.303855 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:21.303847    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:45:21.303909 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:21.303862    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:45:21.303909 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:21.303871    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f4lds for pod openshift-network-diagnostics/network-check-target-tsl56: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:21.303998 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:21.303914    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds podName:9192a864-1460-4306-b8b8-c2b51fabe108 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:29.303901265 +0000 UTC m=+18.158745156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f4lds" (UniqueName: "kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds") pod "network-check-target-tsl56" (UID: "9192a864-1460-4306-b8b8-c2b51fabe108") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:21.523820 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:21.523785    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:21.524005 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:21.523868    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:21.524005 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:21.523882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:21.524079 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:21.524008    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:22.523278 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:22.523233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:22.523838 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:22.523368    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:23.525407 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:23.524661    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:23.525407 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:23.524775    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:23.525918 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:23.525547    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:23.525918 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:23.525672    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:24.523243 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:24.523213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:24.523403 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:24.523329    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:25.523845 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:25.523806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:25.524264 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:25.523947    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:25.524348 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:25.524320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:25.524471 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:25.524424    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:26.523536 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:26.523507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:26.523719 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:26.523631    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:27.523741 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:27.523702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:27.524225 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:27.523845    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:27.524225 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:27.523871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:27.524225 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:27.523986    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:28.524024 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:28.523988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:28.524529 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:28.524113    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:29.269520 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:29.269482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:29.269697 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:29.269570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:29.269697 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:29.269649    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:29.269697 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:29.269668    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:29.269845 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:29.269723    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret podName:e665f43b-b228-44a5-a9ff-412e77074d0d nodeName:}" failed. No retries permitted until 2026-07-28 20:45:45.269702295 +0000 UTC m=+34.124546187 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret") pod "global-pull-secret-syncer-v882q" (UID: "e665f43b-b228-44a5-a9ff-412e77074d0d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:29.269845 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:29.269746    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs podName:81776a2b-e432-46b1-beec-f76495c84ad2 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:45.269735161 +0000 UTC m=+34.124579054 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs") pod "network-metrics-daemon-9j2ft" (UID: "81776a2b-e432-46b1-beec-f76495c84ad2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:29.370941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:29.370908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lds\" (UniqueName: \"kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds\") pod \"network-check-target-tsl56\" (UID: \"9192a864-1460-4306-b8b8-c2b51fabe108\") " pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:29.371150 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:29.371126    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:45:29.371228 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:29.371155    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:45:29.371228 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:29.371169    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f4lds for pod openshift-network-diagnostics/network-check-target-tsl56: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:29.371304 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:29.371233    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds podName:9192a864-1460-4306-b8b8-c2b51fabe108 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:45.371215546 +0000 UTC m=+34.226059455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f4lds" (UniqueName: "kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds") pod "network-check-target-tsl56" (UID: "9192a864-1460-4306-b8b8-c2b51fabe108") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:29.523448 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:29.523365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:29.523448 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:29.523406    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:29.523655 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:29.523502    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:29.523655 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:29.523617    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:30.523733 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:30.523703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:30.524217 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:30.523816    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:31.523754 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:31.523720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:31.524152 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:31.523837    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:31.524152 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:31.523880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:31.524152 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:31.523967    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:32.524111 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:32.524059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:32.524498 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:32.524211    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:33.524154 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:33.524112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:33.524621 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:33.524113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:33.524621 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:33.524250    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:33.524621 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:33.524317    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:34.524160 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.523740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:34.524677 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:34.524261    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:34.626167 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.626054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5" event={"ID":"446e31b8-b81d-461d-8f23-acb33808b1e9","Type":"ContainerStarted","Data":"bfd6a8af765496df6bcae10953552706aaab88fea2dd82935daa82f7b36ad4e4"}
Jul 28 20:45:34.627358 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.627337    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ec92d4e-94e5-4437-96e2-22a783dc73a6" containerID="ff8faf95249035134e9268504f7835a49b655fb66c340c1d309fc7415e8998c2" exitCode=0
Jul 28 20:45:34.627450 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.627400    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vbq" event={"ID":"1ec92d4e-94e5-4437-96e2-22a783dc73a6","Type":"ContainerDied","Data":"ff8faf95249035134e9268504f7835a49b655fb66c340c1d309fc7415e8998c2"}
Jul 28 20:45:34.628901 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.628817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vhls6" event={"ID":"ea587c17-0489-450d-a08b-99232d028831","Type":"ContainerStarted","Data":"3d8e3d918a920f376b0617edaf17ff73c971598f35f4037b3fa8c0f4592541c0"}
Jul 28 20:45:34.630495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.630479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal" event={"ID":"c29c2ef69ffb500d6d735a526717d1b8","Type":"ContainerStarted","Data":"8d15e8576f2f0c9af654338c122072ccb1b1e65d677d59caf1e9df7de9431e6b"}
Jul 28 20:45:34.631891 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.631873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kkgdz" event={"ID":"b4096284-810e-4423-a040-0658e955aceb","Type":"ContainerStarted","Data":"d992d97d3bbfd8ddfd0a11c1200645cf67c8969fc5fafe1f91cbec56bf567931"}
Jul 28 20:45:34.641398 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.641380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c" event={"ID":"a068daf5-405c-4591-8ec2-cc5cb3bd545f","Type":"ContainerStarted","Data":"7642026b5a3f2bea42305a1576d7bd0908bb6264f2b34051322898c5efd9aa84"}
Jul 28 20:45:34.641505 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.641438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c" event={"ID":"a068daf5-405c-4591-8ec2-cc5cb3bd545f","Type":"ContainerStarted","Data":"590b37c84d61d462abb9427d197b1938927f9c8ac838970c0c42939a8d5cebb1"}
Jul 28 20:45:34.641505 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.641484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c" event={"ID":"a068daf5-405c-4591-8ec2-cc5cb3bd545f","Type":"ContainerStarted","Data":"5c972152fab365871e9b581d1f0112745ff5b4860f7ca24294e5104b4a690141"}
Jul 28 20:45:34.641611 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.641523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c" event={"ID":"a068daf5-405c-4591-8ec2-cc5cb3bd545f","Type":"ContainerStarted","Data":"afca6a9c102c86e4a14b70a3eb594cf96ce8280a5269bc5e114c9528f387b2a3"}
Jul 28 20:45:34.642853 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.642831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9wb9" event={"ID":"87a4da6a-0543-4046-b35d-00579314cafe","Type":"ContainerStarted","Data":"31e8917376cab61bc4ba9708bf2ad08cd08c7c11bd4f78c22ee9b69c710bb8ea"}
Jul 28 20:45:34.644296 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.644278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lx58r" event={"ID":"d014f55f-c43a-4ef5-bd1e-e517a72ea8a1","Type":"ContainerStarted","Data":"f64455bbd664444fadded46e066eff76fddd9e39afa7a7561ddd9f3a4f5088ad"}
Jul 28 20:45:34.645702 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.645681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8hrfs" event={"ID":"de6d5b34-f052-4613-a09c-4892d9fac92d","Type":"ContainerStarted","Data":"185905eb35c1f8008612b2e3bb37b81e3e26242c9fa6c86d4a9df3262c3617de"}
Jul 28 20:45:34.693978 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.693934    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r9wb9" podStartSLOduration=3.072579205 podStartE2EDuration="22.693920344s" totalImagesPullingTime="19.621341139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:34.693647965 +0000 UTC m=+23.548491883" watchObservedRunningTime="2026-07-28 20:45:34.693920344 +0000 UTC m=+23.548764255"
Jul 28 20:45:34.694174 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.694041    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-164.ec2.internal" podStartSLOduration=21.694033329 podStartE2EDuration="21.694033329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:34.67077174 +0000 UTC m=+23.525615651" watchObservedRunningTime="2026-07-28 20:45:34.694033329 +0000 UTC m=+23.548877244"
Jul 28 20:45:34.728693 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.728577    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vhls6" podStartSLOduration=3.066379402 podStartE2EDuration="22.72855779s" totalImagesPullingTime="19.662178388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:34.712991701 +0000 UTC m=+23.567835613" watchObservedRunningTime="2026-07-28 20:45:34.72855779 +0000 UTC m=+23.583401700"
Jul 28 20:45:34.745803 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.745750    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lx58r" podStartSLOduration=3.11650161 podStartE2EDuration="22.745729425s" totalImagesPullingTime="19.629227815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:34.745285695 +0000 UTC m=+23.600129606" watchObservedRunningTime="2026-07-28 20:45:34.745729425 +0000 UTC m=+23.600573338"
Jul 28 20:45:34.746387 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.746353    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8hrfs" podStartSLOduration=3.09513648 podStartE2EDuration="22.746338403s" totalImagesPullingTime="19.651201923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:34.729092267 +0000 UTC m=+23.583936165" watchObservedRunningTime="2026-07-28 20:45:34.746338403 +0000 UTC m=+23.601182315"
Jul 28 20:45:34.765543 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:34.765501    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kkgdz" podStartSLOduration=3.074972965 podStartE2EDuration="22.765486381s" totalImagesPullingTime="19.690513416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:34.764893724 +0000 UTC m=+23.619737635" watchObservedRunningTime="2026-07-28 20:45:34.765486381 +0000 UTC m=+23.620330291"
Jul 28 20:45:35.434439 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:35.434409    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 20:45:35.491155 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:35.491038    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T20:45:35.434435786Z","UUID":"2a77bde7-5cc4-4870-9886-af982ebb0548","Handler":null,"Name":"","Endpoint":""}
Jul 28 20:45:35.492627 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:35.492602    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 20:45:35.492764 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:35.492649    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 20:45:35.523891 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:35.523869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:35.523998 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:35.523869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:35.523998 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:35.523980    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:35.524110 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:35.524089    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:35.651871 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:35.651779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c" event={"ID":"a068daf5-405c-4591-8ec2-cc5cb3bd545f","Type":"ContainerStarted","Data":"c860946563d6072c4d33b4c9ffdfb7157b9819fc1aded740457dcdd074d252ac"}
Jul 28 20:45:35.652352 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:35.651876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c" event={"ID":"a068daf5-405c-4591-8ec2-cc5cb3bd545f","Type":"ContainerStarted","Data":"8f2c9be3aefac528668b8bb13786588e31718e543180eee0361d510cd10227ae"}
Jul 28 20:45:35.653389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:35.653366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ftxmm" event={"ID":"cf9fedae-ce04-423f-aefb-979d1cd5ad5a","Type":"ContainerStarted","Data":"a61e59166b04aa60e591bdae5d56b63f1ba3b3533325a9c65cb1152cc729e8b8"}
Jul 28 20:45:35.655482 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:35.655419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5" event={"ID":"446e31b8-b81d-461d-8f23-acb33808b1e9","Type":"ContainerStarted","Data":"fd403762a4141048bed99fb2f89e945a4f709d595f14b59683543584475278bc"}
Jul 28 20:45:36.523943 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:36.523902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:36.524196 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:36.524024    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:37.523443 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:37.523409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:37.523899 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:37.523575    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:37.523899 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:37.523602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:37.523899 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:37.523700    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:37.667553 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:37.667515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c" event={"ID":"a068daf5-405c-4591-8ec2-cc5cb3bd545f","Type":"ContainerStarted","Data":"e8077177094c1a8fe6933672d64a450ccd6854c954e54d3b4f5a52499ed4f94d"}
Jul 28 20:45:37.669662 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:37.669637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5" event={"ID":"446e31b8-b81d-461d-8f23-acb33808b1e9","Type":"ContainerStarted","Data":"e263ab8a640ad8a287b59554845b8acb0ae5b5e7425d5dbbf9a5304b9224d993"}
Jul 28 20:45:37.690801 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:37.690763    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ftxmm" podStartSLOduration=6.038690735 podStartE2EDuration="25.690751289s" totalImagesPullingTime="19.652060554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:35.669670757 +0000 UTC m=+24.524514671" watchObservedRunningTime="2026-07-28 20:45:37.690751289 +0000 UTC m=+26.545595199"
Jul 28 20:45:37.691309 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:37.691279    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9xzq5" podStartSLOduration=3.818567788 podStartE2EDuration="25.691269736s" totalImagesPullingTime="21.872701948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:37.690509734 +0000 UTC m=+26.545353645" watchObservedRunningTime="2026-07-28 20:45:37.691269736 +0000 UTC m=+26.546113648"
Jul 28 20:45:38.524010 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:38.523808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:38.524441 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:38.524111    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:38.620813 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:38.620779    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:38.621526 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:38.621505    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:38.672108 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:38.672038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:38.672339 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:38.672317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lx58r"
Jul 28 20:45:39.523336 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:39.523161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:39.523530 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:39.523181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:39.523530 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:39.523411    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:39.523530 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:39.523512    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:39.676820 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:39.676790    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ec92d4e-94e5-4437-96e2-22a783dc73a6" containerID="bc8577f03445c95640fe85aed7149e59a73525ce922577138aa0b7dd4e59e6c0" exitCode=0
Jul 28 20:45:39.677594 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:39.676870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vbq" event={"ID":"1ec92d4e-94e5-4437-96e2-22a783dc73a6","Type":"ContainerDied","Data":"bc8577f03445c95640fe85aed7149e59a73525ce922577138aa0b7dd4e59e6c0"}
Jul 28 20:45:39.680804 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:39.680779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c" event={"ID":"a068daf5-405c-4591-8ec2-cc5cb3bd545f","Type":"ContainerStarted","Data":"f70bc19e493f7cc795caa493c34cdb6e07a0e6d1d34b4f8bcaffa2816a43ca86"}
Jul 28 20:45:39.736663 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:39.736614    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c" podStartSLOduration=8.04595693 podStartE2EDuration="27.736600379s" totalImagesPullingTime="19.690643449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:39.735274213 +0000 UTC m=+28.590118123" watchObservedRunningTime="2026-07-28 20:45:39.736600379 +0000 UTC m=+28.591444289"
Jul 28 20:45:40.524018 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:40.523985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:40.524273 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:40.524107    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:40.683024 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:40.682993    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 20:45:40.683455 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:40.683431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:40.683513 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:40.683504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:40.699070 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:40.699042    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:40.699386 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:40.699365    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:41.002681 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:41.002655    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v882q"]
Jul 28 20:45:41.002844 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:41.002761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:41.002914 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:41.002858    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:41.007956 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:41.007930    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tsl56"]
Jul 28 20:45:41.008062 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:41.008019    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:41.008149 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:41.008118    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:41.010600 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:41.010580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9j2ft"]
Jul 28 20:45:41.010699 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:41.010665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:41.010759 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:41.010734    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:41.689545 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:41.689510    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ec92d4e-94e5-4437-96e2-22a783dc73a6" containerID="e8f890efdf950fd4c9937f37f4c0b394ae890632376093cd8b33b374e4980ceb" exitCode=0
Jul 28 20:45:41.689913 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:41.689580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vbq" event={"ID":"1ec92d4e-94e5-4437-96e2-22a783dc73a6","Type":"ContainerDied","Data":"e8f890efdf950fd4c9937f37f4c0b394ae890632376093cd8b33b374e4980ceb"}
Jul 28 20:45:41.689913 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:41.689629    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 20:45:41.692820 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:41.692802    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:45:42.524369 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:42.524159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:42.524476 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:42.524167    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:42.524540 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:42.524471    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:42.524596 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:42.524557    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:42.524596 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:42.524197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:42.524681 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:42.524660    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:42.694057 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:42.694029    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ec92d4e-94e5-4437-96e2-22a783dc73a6" containerID="453393cf32c84c38331de822121844e31fc4e22dbf89e36db9ab4064794c176b" exitCode=0
Jul 28 20:45:42.694407 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:42.694111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vbq" event={"ID":"1ec92d4e-94e5-4437-96e2-22a783dc73a6","Type":"ContainerDied","Data":"453393cf32c84c38331de822121844e31fc4e22dbf89e36db9ab4064794c176b"}
Jul 28 20:45:44.523787 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:44.523750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:44.523787 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:44.523782    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:44.524392 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:44.523782    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:44.524392 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:44.523884    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:44.524392 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:44.523959    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:44.524392 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:44.524021    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:45.285834 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:45.285797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:45.286016 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:45.285857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:45.286016 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:45.285957    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:45.286142 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:45.286024    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret podName:e665f43b-b228-44a5-a9ff-412e77074d0d nodeName:}" failed. No retries permitted until 2026-07-28 20:46:17.286007422 +0000 UTC m=+66.140851314 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret") pod "global-pull-secret-syncer-v882q" (UID: "e665f43b-b228-44a5-a9ff-412e77074d0d") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:45:45.286142 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:45.285962    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:45.286142 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:45.286124    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs podName:81776a2b-e432-46b1-beec-f76495c84ad2 nodeName:}" failed. No retries permitted until 2026-07-28 20:46:17.28610515 +0000 UTC m=+66.140949062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs") pod "network-metrics-daemon-9j2ft" (UID: "81776a2b-e432-46b1-beec-f76495c84ad2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:45:45.386694 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:45.386657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lds\" (UniqueName: \"kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds\") pod \"network-check-target-tsl56\" (UID: \"9192a864-1460-4306-b8b8-c2b51fabe108\") " pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:45.386870 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:45.386801    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:45:45.386870 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:45.386833    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:45:45.386870 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:45.386843    2578 projected.go:196] Error preparing data for projected volume kube-api-access-f4lds for pod openshift-network-diagnostics/network-check-target-tsl56: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:45.386997 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:45.386891    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds podName:9192a864-1460-4306-b8b8-c2b51fabe108 nodeName:}" failed. No retries permitted until 2026-07-28 20:46:17.386876164 +0000 UTC m=+66.241720053 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f4lds" (UniqueName: "kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds") pod "network-check-target-tsl56" (UID: "9192a864-1460-4306-b8b8-c2b51fabe108") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:45:46.523361 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:46.523326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:46.523982 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:46.523370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:46.523982 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:46.523336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:46.523982 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:46.523454    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9j2ft" podUID="81776a2b-e432-46b1-beec-f76495c84ad2"
Jul 28 20:45:46.523982 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:46.523524    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsl56" podUID="9192a864-1460-4306-b8b8-c2b51fabe108"
Jul 28 20:45:46.523982 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:46.523599    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v882q" podUID="e665f43b-b228-44a5-a9ff-412e77074d0d"
Jul 28 20:45:47.707643 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:47.707612    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ec92d4e-94e5-4437-96e2-22a783dc73a6" containerID="013f69a7a3d8ed5efa0fedd2e3312854ddd6c237fbaf2aa8ae959a4fdbc61efd" exitCode=0
Jul 28 20:45:47.708124 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:47.707668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vbq" event={"ID":"1ec92d4e-94e5-4437-96e2-22a783dc73a6","Type":"ContainerDied","Data":"013f69a7a3d8ed5efa0fedd2e3312854ddd6c237fbaf2aa8ae959a4fdbc61efd"}
Jul 28 20:45:48.005829 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.005798    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-164.ec2.internal" event="NodeReady"
Jul 28 20:45:48.005965 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.005901    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 20:45:48.059521 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.059491    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6mjf2"]
Jul 28 20:45:48.073863 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.073842    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xmjzf"]
Jul 28 20:45:48.074004 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.073979    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.077019 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.076966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ph6jd\""
Jul 28 20:45:48.077019 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.076998    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 20:45:48.077223 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.077002    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 20:45:48.087674 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.087656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6mjf2"]
Jul 28 20:45:48.087764 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.087686    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xmjzf"]
Jul 28 20:45:48.087764 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.087744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xmjzf"
Jul 28 20:45:48.090461 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.090446    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 20:45:48.090587 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.090569    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kmbt4\""
Jul 28 20:45:48.091033 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.091017    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 20:45:48.091143 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.091043    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 20:45:48.206301 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.206276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-config-volume\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.206301 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.206307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.206495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.206325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acb3e3c8-2af8-4dad-a317-c04e4515711f-cert\") pod \"ingress-canary-xmjzf\" (UID: \"acb3e3c8-2af8-4dad-a317-c04e4515711f\") " pod="openshift-ingress-canary/ingress-canary-xmjzf"
Jul 28 20:45:48.206495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.206342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lks6x\" (UniqueName: \"kubernetes.io/projected/acb3e3c8-2af8-4dad-a317-c04e4515711f-kube-api-access-lks6x\") pod \"ingress-canary-xmjzf\" (UID: \"acb3e3c8-2af8-4dad-a317-c04e4515711f\") " pod="openshift-ingress-canary/ingress-canary-xmjzf"
Jul 28 20:45:48.206495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.206431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-tmp-dir\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.206495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.206468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-278gj\" (UniqueName: \"kubernetes.io/projected/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-kube-api-access-278gj\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.307133 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.307103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.307133 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.307133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acb3e3c8-2af8-4dad-a317-c04e4515711f-cert\") pod \"ingress-canary-xmjzf\" (UID: \"acb3e3c8-2af8-4dad-a317-c04e4515711f\") " pod="openshift-ingress-canary/ingress-canary-xmjzf"
Jul 28 20:45:48.307346 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.307153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lks6x\" (UniqueName: \"kubernetes.io/projected/acb3e3c8-2af8-4dad-a317-c04e4515711f-kube-api-access-lks6x\") pod \"ingress-canary-xmjzf\" (UID: \"acb3e3c8-2af8-4dad-a317-c04e4515711f\") " pod="openshift-ingress-canary/ingress-canary-xmjzf"
Jul 28 20:45:48.307346 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.307206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-tmp-dir\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.307346 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:48.307245    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:45:48.307346 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.307250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-278gj\" (UniqueName: \"kubernetes.io/projected/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-kube-api-access-278gj\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.307346 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.307290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-config-volume\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.307346 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:48.307335    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls podName:cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:48.807316831 +0000 UTC m=+37.662160729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls") pod "dns-default-6mjf2" (UID: "cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961") : secret "dns-default-metrics-tls" not found
Jul 28 20:45:48.307603 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.307549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-tmp-dir\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.307818 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.307799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-config-volume\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.311442 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.311419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acb3e3c8-2af8-4dad-a317-c04e4515711f-cert\") pod \"ingress-canary-xmjzf\" (UID: \"acb3e3c8-2af8-4dad-a317-c04e4515711f\") " pod="openshift-ingress-canary/ingress-canary-xmjzf"
Jul 28 20:45:48.321086 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.321052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-278gj\" (UniqueName: \"kubernetes.io/projected/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-kube-api-access-278gj\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.324678 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.324655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lks6x\" (UniqueName: \"kubernetes.io/projected/acb3e3c8-2af8-4dad-a317-c04e4515711f-kube-api-access-lks6x\") pod \"ingress-canary-xmjzf\" (UID: \"acb3e3c8-2af8-4dad-a317-c04e4515711f\") " pod="openshift-ingress-canary/ingress-canary-xmjzf"
Jul 28 20:45:48.395552 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.395486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xmjzf"
Jul 28 20:45:48.524002 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.523846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:45:48.524151 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.523846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:45:48.524204 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.523846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:45:48.526661 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.526638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xmjzf"]
Jul 28 20:45:48.527225 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.527207    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5747k\""
Jul 28 20:45:48.527225 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.527217    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 20:45:48.527558 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.527545    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 20:45:48.527611 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.527600    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 20:45:48.527788 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.527772    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2qcwd\""
Jul 28 20:45:48.527997 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.527980    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 20:45:48.530973 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:48.530954    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb3e3c8_2af8_4dad_a317_c04e4515711f.slice/crio-3909d3aea033b9f789fa1eb2debc9b0ecfa5d56c45c65323f3e980a9f2e9dcca WatchSource:0}: Error finding container 3909d3aea033b9f789fa1eb2debc9b0ecfa5d56c45c65323f3e980a9f2e9dcca: Status 404 returned error can't find the container with id 3909d3aea033b9f789fa1eb2debc9b0ecfa5d56c45c65323f3e980a9f2e9dcca
Jul 28 20:45:48.695517 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.695415    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kg2x9"]
Jul 28 20:45:48.698799 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.698785    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.702642 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.702618    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 20:45:48.702748 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.702650    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 20:45:48.702748 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.702687    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 20:45:48.703266 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.703244    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 20:45:48.703843 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.703823    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nwhvj\""
Jul 28 20:45:48.711378 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.711355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xmjzf" event={"ID":"acb3e3c8-2af8-4dad-a317-c04e4515711f","Type":"ContainerStarted","Data":"3909d3aea033b9f789fa1eb2debc9b0ecfa5d56c45c65323f3e980a9f2e9dcca"}
Jul 28 20:45:48.714331 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.714307    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ec92d4e-94e5-4437-96e2-22a783dc73a6" containerID="224e9663f3c16cc288dc5493c1209615c0536a0c09ce0c793b2e23b62d7280d6" exitCode=0
Jul 28 20:45:48.714427 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.714339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vbq" event={"ID":"1ec92d4e-94e5-4437-96e2-22a783dc73a6","Type":"ContainerDied","Data":"224e9663f3c16cc288dc5493c1209615c0536a0c09ce0c793b2e23b62d7280d6"}
Jul 28 20:45:48.717396 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.717377    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kg2x9"]
Jul 28 20:45:48.810089 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.810052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:48.810181 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.810101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/349dc143-219c-4d79-a6a3-24778d73a200-data-volume\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.810219 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:48.810191    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:45:48.810252 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.810215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/349dc143-219c-4d79-a6a3-24778d73a200-crio-socket\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.810252 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:48.810236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls podName:cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:49.810222081 +0000 UTC m=+38.665065970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls") pod "dns-default-6mjf2" (UID: "cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961") : secret "dns-default-metrics-tls" not found
Jul 28 20:45:48.810316 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.810264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.810356 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.810338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbq6\" (UniqueName: \"kubernetes.io/projected/349dc143-219c-4d79-a6a3-24778d73a200-kube-api-access-kdbq6\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.810536 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.810516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.911557 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.911523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.911689 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.911574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/349dc143-219c-4d79-a6a3-24778d73a200-data-volume\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.911689 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:48.911643    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:45:48.911773 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.911693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/349dc143-219c-4d79-a6a3-24778d73a200-crio-socket\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.911773 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:48.911718    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:49.411699117 +0000 UTC m=+38.266543023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : configmap "kube-rbac-proxy" not found
Jul 28 20:45:48.911773 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.911751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.911899 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.911815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbq6\" (UniqueName: \"kubernetes.io/projected/349dc143-219c-4d79-a6a3-24778d73a200-kube-api-access-kdbq6\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.911899 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.911855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/349dc143-219c-4d79-a6a3-24778d73a200-data-volume\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.911899 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.911886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/349dc143-219c-4d79-a6a3-24778d73a200-crio-socket\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:48.912032 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:48.911915    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:45:48.912032 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:48.911995    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:49.411976508 +0000 UTC m=+38.266820413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:45:48.921925 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:48.921904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbq6\" (UniqueName: \"kubernetes.io/projected/349dc143-219c-4d79-a6a3-24778d73a200-kube-api-access-kdbq6\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:49.415776 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.415743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:49.415943 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.415838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:49.415943 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:49.415862    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:45:49.415943 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:49.415923    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:50.415903792 +0000 UTC m=+39.270747681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : configmap "kube-rbac-proxy" not found
Jul 28 20:45:49.416137 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:49.415968    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:45:49.416137 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:49.416015    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:50.416002201 +0000 UTC m=+39.270846090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:45:49.695337 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.695254    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv"]
Jul 28 20:45:49.698111 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.698069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv"
Jul 28 20:45:49.700768 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.700642    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 20:45:49.700768 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.700656    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n8dvm\""
Jul 28 20:45:49.702367 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.701646    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 20:45:49.710110 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.709916    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv"]
Jul 28 20:45:49.721067 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.721039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94vbq" event={"ID":"1ec92d4e-94e5-4437-96e2-22a783dc73a6","Type":"ContainerStarted","Data":"33ab9ba84ec196a4790b3d8233fc65d7d984f4f0ae0b0d947b823403b7026646"}
Jul 28 20:45:49.794378 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.794324    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-94vbq" podStartSLOduration=7.56681718 podStartE2EDuration="37.794308209s" totalImagesPullingTime="30.227491029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:49.793295528 +0000 UTC m=+38.648139459" watchObservedRunningTime="2026-07-28 20:45:49.794308209 +0000 UTC m=+38.649152120"
Jul 28 20:45:49.818521 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.818481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4962t\" (UniqueName: \"kubernetes.io/projected/69070dc3-cb63-4714-9542-f866288990d1-kube-api-access-4962t\") pod \"migrator-6865c9967-zbxnv\" (UID: \"69070dc3-cb63-4714-9542-f866288990d1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv"
Jul 28 20:45:49.818685 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.818631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:49.818767 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:49.818743    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:45:49.818892 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:49.818819    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls podName:cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:51.818792093 +0000 UTC m=+40.673635985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls") pod "dns-default-6mjf2" (UID: "cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961") : secret "dns-default-metrics-tls" not found
Jul 28 20:45:49.919384 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.919345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4962t\" (UniqueName: \"kubernetes.io/projected/69070dc3-cb63-4714-9542-f866288990d1-kube-api-access-4962t\") pod \"migrator-6865c9967-zbxnv\" (UID: \"69070dc3-cb63-4714-9542-f866288990d1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv"
Jul 28 20:45:49.931346 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:49.931322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4962t\" (UniqueName: \"kubernetes.io/projected/69070dc3-cb63-4714-9542-f866288990d1-kube-api-access-4962t\") pod \"migrator-6865c9967-zbxnv\" (UID: \"69070dc3-cb63-4714-9542-f866288990d1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv"
Jul 28 20:45:50.013451 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:50.013345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv"
Jul 28 20:45:50.423488 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:50.423400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:50.423488 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:50.423472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:50.423731 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:50.423540    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:45:50.423731 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:50.423588    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:45:50.423731 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:50.423613    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:52.42359551 +0000 UTC m=+41.278439402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : configmap "kube-rbac-proxy" not found
Jul 28 20:45:50.423731 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:50.423632    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:52.42362096 +0000 UTC m=+41.278464848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:45:50.763462 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:50.763434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv"]
Jul 28 20:45:50.767566 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:50.767539    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69070dc3_cb63_4714_9542_f866288990d1.slice/crio-28125a2b63567c0126e86f79b84375b7faafd7f59e766d13f19e86fcc6742ba3 WatchSource:0}: Error finding container 28125a2b63567c0126e86f79b84375b7faafd7f59e766d13f19e86fcc6742ba3: Status 404 returned error can't find the container with id 28125a2b63567c0126e86f79b84375b7faafd7f59e766d13f19e86fcc6742ba3
Jul 28 20:45:50.898382 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:50.898359    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9wb9_87a4da6a-0543-4046-b35d-00579314cafe/dns-node-resolver/0.log"
Jul 28 20:45:51.494150 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:51.494070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8hrfs_de6d5b34-f052-4613-a09c-4892d9fac92d/node-ca/0.log"
Jul 28 20:45:51.728591 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:51.728358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xmjzf" event={"ID":"acb3e3c8-2af8-4dad-a317-c04e4515711f","Type":"ContainerStarted","Data":"5c99a843b96ce8bc9a721dec8a22fbf225350171bbcf5546149200928658a153"}
Jul 28 20:45:51.730286 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:51.730254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv" event={"ID":"69070dc3-cb63-4714-9542-f866288990d1","Type":"ContainerStarted","Data":"28125a2b63567c0126e86f79b84375b7faafd7f59e766d13f19e86fcc6742ba3"}
Jul 28 20:45:51.747809 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:51.747708    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xmjzf" podStartSLOduration=1.5950412950000001 podStartE2EDuration="3.747689379s" totalImagesPullingTime="2.152648084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:51.746738164 +0000 UTC m=+40.601582077" watchObservedRunningTime="2026-07-28 20:45:51.747689379 +0000 UTC m=+40.602533292"
Jul 28 20:45:51.834451 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:51.834422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:51.834918 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:51.834581    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:45:51.834918 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:51.834661    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls podName:cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:55.834639955 +0000 UTC m=+44.689483860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls") pod "dns-default-6mjf2" (UID: "cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961") : secret "dns-default-metrics-tls" not found
Jul 28 20:45:52.140927 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.140819    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-dwf49"]
Jul 28 20:45:52.143616 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.143600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.146272 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.146243    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 20:45:52.146403 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.146340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 20:45:52.146477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.146463    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g22m9\""
Jul 28 20:45:52.146554 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.146535    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 20:45:52.146594 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.146578    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 20:45:52.152953 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.152924    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-dwf49"]
Jul 28 20:45:52.238142 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.238106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/82835e66-4c68-4b9b-8969-427a8cece631-signing-key\") pod \"service-ca-658d7b7c78-dwf49\" (UID: \"82835e66-4c68-4b9b-8969-427a8cece631\") " pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.238142 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.238142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjlp\" (UniqueName: \"kubernetes.io/projected/82835e66-4c68-4b9b-8969-427a8cece631-kube-api-access-sfjlp\") pod \"service-ca-658d7b7c78-dwf49\" (UID: \"82835e66-4c68-4b9b-8969-427a8cece631\") " pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.238376 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.238185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/82835e66-4c68-4b9b-8969-427a8cece631-signing-cabundle\") pod \"service-ca-658d7b7c78-dwf49\" (UID: \"82835e66-4c68-4b9b-8969-427a8cece631\") " pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.338611 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.338575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/82835e66-4c68-4b9b-8969-427a8cece631-signing-key\") pod \"service-ca-658d7b7c78-dwf49\" (UID: \"82835e66-4c68-4b9b-8969-427a8cece631\") " pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.338611 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.338610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjlp\" (UniqueName: \"kubernetes.io/projected/82835e66-4c68-4b9b-8969-427a8cece631-kube-api-access-sfjlp\") pod \"service-ca-658d7b7c78-dwf49\" (UID: \"82835e66-4c68-4b9b-8969-427a8cece631\") " pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.338794 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.338643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/82835e66-4c68-4b9b-8969-427a8cece631-signing-cabundle\") pod \"service-ca-658d7b7c78-dwf49\" (UID: \"82835e66-4c68-4b9b-8969-427a8cece631\") " pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.339373 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.339356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/82835e66-4c68-4b9b-8969-427a8cece631-signing-cabundle\") pod \"service-ca-658d7b7c78-dwf49\" (UID: \"82835e66-4c68-4b9b-8969-427a8cece631\") " pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.341790 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.341760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/82835e66-4c68-4b9b-8969-427a8cece631-signing-key\") pod \"service-ca-658d7b7c78-dwf49\" (UID: \"82835e66-4c68-4b9b-8969-427a8cece631\") " pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.348963 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.348933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjlp\" (UniqueName: \"kubernetes.io/projected/82835e66-4c68-4b9b-8969-427a8cece631-kube-api-access-sfjlp\") pod \"service-ca-658d7b7c78-dwf49\" (UID: \"82835e66-4c68-4b9b-8969-427a8cece631\") " pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.439362 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.439290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:52.439362 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.439349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:52.439508 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:52.439404    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:45:52.439508 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:52.439451    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:45:52.439508 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:52.439465    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:56.439448571 +0000 UTC m=+45.294292461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : configmap "kube-rbac-proxy" not found
Jul 28 20:45:52.439508 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:52.439488    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:45:56.439477875 +0000 UTC m=+45.294321765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:45:52.452877 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.452859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-dwf49"
Jul 28 20:45:52.573951 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.573808    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-dwf49"]
Jul 28 20:45:52.576177 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:45:52.576154    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82835e66_4c68_4b9b_8969_427a8cece631.slice/crio-ea6c867a93081adec509d5b0bcdbbfa6a6681d9f9dcad6146170a33d977dfcea WatchSource:0}: Error finding container ea6c867a93081adec509d5b0bcdbbfa6a6681d9f9dcad6146170a33d977dfcea: Status 404 returned error can't find the container with id ea6c867a93081adec509d5b0bcdbbfa6a6681d9f9dcad6146170a33d977dfcea
Jul 28 20:45:52.694486 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.694412    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xmjzf_acb3e3c8-2af8-4dad-a317-c04e4515711f/serve-healthcheck-canary/0.log"
Jul 28 20:45:52.733263 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.733215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-dwf49" event={"ID":"82835e66-4c68-4b9b-8969-427a8cece631","Type":"ContainerStarted","Data":"ea6c867a93081adec509d5b0bcdbbfa6a6681d9f9dcad6146170a33d977dfcea"}
Jul 28 20:45:52.734819 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.734788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv" event={"ID":"69070dc3-cb63-4714-9542-f866288990d1","Type":"ContainerStarted","Data":"f73585dc8a119202cca814fd61de2c2f9dd49bf16a1b92988eac125c97c127dd"}
Jul 28 20:45:52.734923 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.734840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv" event={"ID":"69070dc3-cb63-4714-9542-f866288990d1","Type":"ContainerStarted","Data":"057656a30db6fd2166faf5a8c47da5a795021eca850bc627c54604f6d727a62a"}
Jul 28 20:45:52.757176 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:52.757130    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zbxnv" podStartSLOduration=2.639747878 podStartE2EDuration="3.757116513s" totalImagesPullingTime="1.117368635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:52.756463014 +0000 UTC m=+41.611306926" watchObservedRunningTime="2026-07-28 20:45:52.757116513 +0000 UTC m=+41.611960427"
Jul 28 20:45:55.744128 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:55.744069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-dwf49" event={"ID":"82835e66-4c68-4b9b-8969-427a8cece631","Type":"ContainerStarted","Data":"52a54403227cded1e5ca2a40b50c8d5b27bcc3f4581eeea0d8273dc2d6cf904e"}
Jul 28 20:45:55.865749 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:55.865713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:45:55.865941 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:55.865849    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:45:55.865941 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:55.865921    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls podName:cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961 nodeName:}" failed. No retries permitted until 2026-07-28 20:46:03.86590316 +0000 UTC m=+52.720747055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls") pod "dns-default-6mjf2" (UID: "cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961") : secret "dns-default-metrics-tls" not found
Jul 28 20:45:56.471784 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:56.471744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:56.471983 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:45:56.471840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:45:56.471983 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:56.471868    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:45:56.471983 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:56.471946    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:46:04.471926147 +0000 UTC m=+53.326770052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : configmap "kube-rbac-proxy" not found
Jul 28 20:45:56.472186 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:56.472005    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:45:56.472186 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:45:56.472054    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:46:04.472038945 +0000 UTC m=+53.326882849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:46:03.932973 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:03.932930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:46:03.935358 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:03.935338    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961-metrics-tls\") pod \"dns-default-6mjf2\" (UID: \"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961\") " pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:46:03.983511 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:03.983478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:46:04.128523 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:04.128450    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-dwf49" podStartSLOduration=9.835330047 podStartE2EDuration="12.128429188s" totalImagesPullingTime="2.293099141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:55.766091321 +0000 UTC m=+44.620935231" watchObservedRunningTime="2026-07-28 20:46:04.128429188 +0000 UTC m=+52.983273099"
Jul 28 20:46:04.129343 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:04.129317    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6mjf2"]
Jul 28 20:46:04.132680 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:04.132649    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7e9d2c_fc43_40e4_9412_1a2f1f8f0961.slice/crio-9be5af0f748fa78f08a2d5b32e1672adc681c43480344b4a89be2e25d28e1f32 WatchSource:0}: Error finding container 9be5af0f748fa78f08a2d5b32e1672adc681c43480344b4a89be2e25d28e1f32: Status 404 returned error can't find the container with id 9be5af0f748fa78f08a2d5b32e1672adc681c43480344b4a89be2e25d28e1f32
Jul 28 20:46:04.538468 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:04.538415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:46:04.538655 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:04.538513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:46:04.538701 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:46:04.538669    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:46:04.538779 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:46:04.538767    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls podName:349dc143-219c-4d79-a6a3-24778d73a200 nodeName:}" failed. No retries permitted until 2026-07-28 20:46:20.538742365 +0000 UTC m=+69.393586274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg2x9" (UID: "349dc143-219c-4d79-a6a3-24778d73a200") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:46:04.539006 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:04.538988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/349dc143-219c-4d79-a6a3-24778d73a200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:46:04.769732 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:04.769687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6mjf2" event={"ID":"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961","Type":"ContainerStarted","Data":"9be5af0f748fa78f08a2d5b32e1672adc681c43480344b4a89be2e25d28e1f32"}
Jul 28 20:46:06.777267 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:06.777229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6mjf2" event={"ID":"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961","Type":"ContainerStarted","Data":"252c76178ce1cdd3075f4e3c43a0516f2dd75cef17e19550d4c07f1c36060500"}
Jul 28 20:46:06.777741 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:06.777330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6mjf2" event={"ID":"cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961","Type":"ContainerStarted","Data":"f3c60ca816831289c03c8c14dd1f930968150448acef82b2db9aae96026429fd"}
Jul 28 20:46:06.777741 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:06.777371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:46:06.799227 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:06.799187    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6mjf2" podStartSLOduration=16.841441026 podStartE2EDuration="18.799172977s" totalImagesPullingTime="1.957731951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:06.798575608 +0000 UTC m=+55.653419500" watchObservedRunningTime="2026-07-28 20:46:06.799172977 +0000 UTC m=+55.654016888"
Jul 28 20:46:12.908310 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:12.908275    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-vr4rz"]
Jul 28 20:46:12.934416 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:12.934384    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-vr4rz"]
Jul 28 20:46:12.934596 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:12.934516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-vr4rz"
Jul 28 20:46:12.937539 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:12.937523    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xz9gd\""
Jul 28 20:46:12.937823 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:12.937804    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 20:46:12.938749 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:12.938736    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 20:46:12.994384 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:12.994347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rxj9\" (UniqueName: \"kubernetes.io/projected/98b94339-faa6-414e-be68-c6ec91e9b4eb-kube-api-access-9rxj9\") pod \"downloads-7c97dc4555-vr4rz\" (UID: \"98b94339-faa6-414e-be68-c6ec91e9b4eb\") " pod="openshift-console/downloads-7c97dc4555-vr4rz"
Jul 28 20:46:12.996978 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:12.996947    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65c598f646-wbnmg"]
Jul 28 20:46:13.016723 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.016693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65c598f646-wbnmg"]
Jul 28 20:46:13.016873 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.016796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.019873 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.019848    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 20:46:13.020163 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.020145    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 20:46:13.020382 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.020319    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kskpf\""
Jul 28 20:46:13.020561 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.020456    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 20:46:13.031095 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.031063    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 20:46:13.095637 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.095605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-registry-tls\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.095637 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.095637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng4v4\" (UniqueName: \"kubernetes.io/projected/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-kube-api-access-ng4v4\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.095824 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.095655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-bound-sa-token\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.095824 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.095678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rxj9\" (UniqueName: \"kubernetes.io/projected/98b94339-faa6-414e-be68-c6ec91e9b4eb-kube-api-access-9rxj9\") pod \"downloads-7c97dc4555-vr4rz\" (UID: \"98b94339-faa6-414e-be68-c6ec91e9b4eb\") " pod="openshift-console/downloads-7c97dc4555-vr4rz"
Jul 28 20:46:13.095824 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.095694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-ca-trust-extracted\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.095824 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.095778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-trusted-ca\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.095967 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.095831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-image-registry-private-configuration\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.095967 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.095875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-registry-certificates\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.095967 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.095893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-installation-pull-secrets\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.139876 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.139848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rxj9\" (UniqueName: \"kubernetes.io/projected/98b94339-faa6-414e-be68-c6ec91e9b4eb-kube-api-access-9rxj9\") pod \"downloads-7c97dc4555-vr4rz\" (UID: \"98b94339-faa6-414e-be68-c6ec91e9b4eb\") " pod="openshift-console/downloads-7c97dc4555-vr4rz"
Jul 28 20:46:13.196687 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.196622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-trusted-ca\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.196687 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.196665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-image-registry-private-configuration\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.196869 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.196690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-registry-certificates\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.196869 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.196707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-installation-pull-secrets\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.196869 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.196726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-registry-tls\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.196869 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.196743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4v4\" (UniqueName: \"kubernetes.io/projected/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-kube-api-access-ng4v4\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.196869 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.196768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-bound-sa-token\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.196869 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.196796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-ca-trust-extracted\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.197212 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.197190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-ca-trust-extracted\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.197700 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.197601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-trusted-ca\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.197700 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.197601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-registry-certificates\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.199358 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.199330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-registry-tls\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.199452 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.199355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-installation-pull-secrets\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.199942 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.199918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-image-registry-private-configuration\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.219768 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.219732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4v4\" (UniqueName: \"kubernetes.io/projected/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-kube-api-access-ng4v4\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.220372 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.220354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0bb1835-824f-4d59-abe5-5bd6b74e8b59-bound-sa-token\") pod \"image-registry-65c598f646-wbnmg\" (UID: \"b0bb1835-824f-4d59-abe5-5bd6b74e8b59\") " pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.243464 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.243441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-vr4rz"
Jul 28 20:46:13.326409 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.326384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.370829 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.370760    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-vr4rz"]
Jul 28 20:46:13.375914 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:13.375851    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b94339_faa6_414e_be68_c6ec91e9b4eb.slice/crio-bbe4d45711eb54c8fd76bf1c01115eb5e609a857028e80578fbf92cbea19d12e WatchSource:0}: Error finding container bbe4d45711eb54c8fd76bf1c01115eb5e609a857028e80578fbf92cbea19d12e: Status 404 returned error can't find the container with id bbe4d45711eb54c8fd76bf1c01115eb5e609a857028e80578fbf92cbea19d12e
Jul 28 20:46:13.457188 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.457115    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65c598f646-wbnmg"]
Jul 28 20:46:13.465159 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:13.465135    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bb1835_824f_4d59_abe5_5bd6b74e8b59.slice/crio-8c95f00314bed4fb47920f10b84f2f9206d67d2c5abc5f90ffeab5a3695420ef WatchSource:0}: Error finding container 8c95f00314bed4fb47920f10b84f2f9206d67d2c5abc5f90ffeab5a3695420ef: Status 404 returned error can't find the container with id 8c95f00314bed4fb47920f10b84f2f9206d67d2c5abc5f90ffeab5a3695420ef
Jul 28 20:46:13.712385 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.712315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5m6c"
Jul 28 20:46:13.802513 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.802483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65c598f646-wbnmg" event={"ID":"b0bb1835-824f-4d59-abe5-5bd6b74e8b59","Type":"ContainerStarted","Data":"5a3b3a7f3fb682d9f38ec3c213b9eba7e7a0f083a338f2ce89506937a629ae2c"}
Jul 28 20:46:13.802685 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.802533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65c598f646-wbnmg" event={"ID":"b0bb1835-824f-4d59-abe5-5bd6b74e8b59","Type":"ContainerStarted","Data":"8c95f00314bed4fb47920f10b84f2f9206d67d2c5abc5f90ffeab5a3695420ef"}
Jul 28 20:46:13.802685 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.802588    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:13.804150 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.804124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-vr4rz" event={"ID":"98b94339-faa6-414e-be68-c6ec91e9b4eb","Type":"ContainerStarted","Data":"bbe4d45711eb54c8fd76bf1c01115eb5e609a857028e80578fbf92cbea19d12e"}
Jul 28 20:46:13.836953 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:13.836762    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65c598f646-wbnmg" podStartSLOduration=1.836739892 podStartE2EDuration="1.836739892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:13.834184045 +0000 UTC m=+62.689027956" watchObservedRunningTime="2026-07-28 20:46:13.836739892 +0000 UTC m=+62.691583783"
Jul 28 20:46:16.782812 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:16.782775    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6mjf2"
Jul 28 20:46:17.327243 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.327206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:46:17.327442 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.327277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:46:17.331218 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.331192    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 20:46:17.332018 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.331999    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 20:46:17.341028 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.341002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e665f43b-b228-44a5-a9ff-412e77074d0d-original-pull-secret\") pod \"global-pull-secret-syncer-v882q\" (UID: \"e665f43b-b228-44a5-a9ff-412e77074d0d\") " pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:46:17.341162 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.341017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81776a2b-e432-46b1-beec-f76495c84ad2-metrics-certs\") pod \"network-metrics-daemon-9j2ft\" (UID: \"81776a2b-e432-46b1-beec-f76495c84ad2\") " pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:46:17.347585 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.347562    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2qcwd\""
Jul 28 20:46:17.355307 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.355291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9j2ft"
Jul 28 20:46:17.428722 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.428261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lds\" (UniqueName: \"kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds\") pod \"network-check-target-tsl56\" (UID: \"9192a864-1460-4306-b8b8-c2b51fabe108\") " pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:46:17.431379 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.431357    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 20:46:17.441471 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.441447    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 20:46:17.454742 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.454686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lds\" (UniqueName: \"kubernetes.io/projected/9192a864-1460-4306-b8b8-c2b51fabe108-kube-api-access-f4lds\") pod \"network-check-target-tsl56\" (UID: \"9192a864-1460-4306-b8b8-c2b51fabe108\") " pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:46:17.496020 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.495992    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9j2ft"]
Jul 28 20:46:17.499122 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:17.499096    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81776a2b_e432_46b1_beec_f76495c84ad2.slice/crio-93fb223b97fb35035230c4687fd3a17cf5a474174e10a186f27e0d7f1bed2fe7 WatchSource:0}: Error finding container 93fb223b97fb35035230c4687fd3a17cf5a474174e10a186f27e0d7f1bed2fe7: Status 404 returned error can't find the container with id 93fb223b97fb35035230c4687fd3a17cf5a474174e10a186f27e0d7f1bed2fe7
Jul 28 20:46:17.637754 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.637665    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5747k\""
Jul 28 20:46:17.640723 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.640704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v882q"
Jul 28 20:46:17.645686 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.645663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:46:17.795840 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.795807    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tsl56"]
Jul 28 20:46:17.801441 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:17.801407    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9192a864_1460_4306_b8b8_c2b51fabe108.slice/crio-26132417e7154785c9c84819a8790193c2426cd2115d46712e08563cbba47d7d WatchSource:0}: Error finding container 26132417e7154785c9c84819a8790193c2426cd2115d46712e08563cbba47d7d: Status 404 returned error can't find the container with id 26132417e7154785c9c84819a8790193c2426cd2115d46712e08563cbba47d7d
Jul 28 20:46:17.814451 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.814390    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v882q"]
Jul 28 20:46:17.817666 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:17.817420    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode665f43b_b228_44a5_a9ff_412e77074d0d.slice/crio-b7a8ef6ea124763633bc8438691df3a0358d94ff26072ea3e01a82a0a7c7b16b WatchSource:0}: Error finding container b7a8ef6ea124763633bc8438691df3a0358d94ff26072ea3e01a82a0a7c7b16b: Status 404 returned error can't find the container with id b7a8ef6ea124763633bc8438691df3a0358d94ff26072ea3e01a82a0a7c7b16b
Jul 28 20:46:17.819393 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.819357    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9j2ft" event={"ID":"81776a2b-e432-46b1-beec-f76495c84ad2","Type":"ContainerStarted","Data":"93fb223b97fb35035230c4687fd3a17cf5a474174e10a186f27e0d7f1bed2fe7"}
Jul 28 20:46:17.820885 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:17.820792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tsl56" event={"ID":"9192a864-1460-4306-b8b8-c2b51fabe108","Type":"ContainerStarted","Data":"26132417e7154785c9c84819a8790193c2426cd2115d46712e08563cbba47d7d"}
Jul 28 20:46:18.827579 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:18.827540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v882q" event={"ID":"e665f43b-b228-44a5-a9ff-412e77074d0d","Type":"ContainerStarted","Data":"b7a8ef6ea124763633bc8438691df3a0358d94ff26072ea3e01a82a0a7c7b16b"}
Jul 28 20:46:18.829941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:18.829856    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9j2ft" event={"ID":"81776a2b-e432-46b1-beec-f76495c84ad2","Type":"ContainerStarted","Data":"71802fbe7f1e707e8e7f049a5e163f34ee899a98c459b84c160e8b646382509f"}
Jul 28 20:46:18.829941 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:18.829910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9j2ft" event={"ID":"81776a2b-e432-46b1-beec-f76495c84ad2","Type":"ContainerStarted","Data":"094bacb8c162c08a0cc360d2dbb382537ce0f8c6910b49f5b3e457529c1d55ef"}
Jul 28 20:46:18.848257 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:18.848209    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9j2ft" podStartSLOduration=65.927133633 podStartE2EDuration="1m6.848191714s" totalImagesPullingTime="921.058081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:18.847492053 +0000 UTC m=+67.702335966" watchObservedRunningTime="2026-07-28 20:46:18.848191714 +0000 UTC m=+67.703035626"
Jul 28 20:46:20.552506 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:20.552427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:46:20.555026 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:20.554998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/349dc143-219c-4d79-a6a3-24778d73a200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg2x9\" (UID: \"349dc143-219c-4d79-a6a3-24778d73a200\") " pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:46:20.809569 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:20.809482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nwhvj\""
Jul 28 20:46:20.816980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:20.816955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kg2x9"
Jul 28 20:46:23.514403 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:23.514376    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kg2x9"]
Jul 28 20:46:23.517092 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:23.517049    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349dc143_219c_4d79_a6a3_24778d73a200.slice/crio-1385426ccbd25f77fbb5984fdada6d2923048308a77e84a017d179fae21c46e9 WatchSource:0}: Error finding container 1385426ccbd25f77fbb5984fdada6d2923048308a77e84a017d179fae21c46e9: Status 404 returned error can't find the container with id 1385426ccbd25f77fbb5984fdada6d2923048308a77e84a017d179fae21c46e9
Jul 28 20:46:23.851219 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:23.851181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tsl56" event={"ID":"9192a864-1460-4306-b8b8-c2b51fabe108","Type":"ContainerStarted","Data":"f70baa9016d6346b6c3f1fc7b3bb58c398173a9787a467f96433268eabb06626"}
Jul 28 20:46:23.851394 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:23.851288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:46:23.852537 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:23.852508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kg2x9" event={"ID":"349dc143-219c-4d79-a6a3-24778d73a200","Type":"ContainerStarted","Data":"1385426ccbd25f77fbb5984fdada6d2923048308a77e84a017d179fae21c46e9"}
Jul 28 20:46:23.854050 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:23.854003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v882q" event={"ID":"e665f43b-b228-44a5-a9ff-412e77074d0d","Type":"ContainerStarted","Data":"21a2356e3cdc67065036cbd5c6d19fa0fb7d4d25d6a20bf92702bcc0bd53a6e3"}
Jul 28 20:46:23.871858 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:23.871809    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tsl56" podStartSLOduration=66.313404888 podStartE2EDuration="1m11.871792267s" totalImagesPullingTime="5.558387379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:23.871285182 +0000 UTC m=+72.726129094" watchObservedRunningTime="2026-07-28 20:46:23.871792267 +0000 UTC m=+72.726636179"
Jul 28 20:46:23.888450 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:23.888409    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v882q" podStartSLOduration=66.335945501 podStartE2EDuration="1m11.88839359s" totalImagesPullingTime="5.552448089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:23.887734477 +0000 UTC m=+72.742578389" watchObservedRunningTime="2026-07-28 20:46:23.88839359 +0000 UTC m=+72.743237506"
Jul 28 20:46:27.945635 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.945438    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"]
Jul 28 20:46:27.951106 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.951057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:27.960665 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.953612    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 20:46:27.960665 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.953888    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 20:46:27.960665 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.954800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9jw7d\""
Jul 28 20:46:27.960665 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.955169    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 20:46:27.960665 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.955338    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 20:46:27.960665 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.955446    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 20:46:27.965300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.965169    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"]
Jul 28 20:46:27.982647 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.982623    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6xrrf"]
Jul 28 20:46:27.986450 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.986432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:27.989474 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.989455    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tsvt2\""
Jul 28 20:46:27.992562 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.989753    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 20:46:27.992562 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.989778    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 20:46:27.992562 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:27.990920    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 20:46:28.006876 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.006855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrm6\" (UniqueName: \"kubernetes.io/projected/7b3f5c6a-d31b-415f-9166-af11041f6afc-kube-api-access-wmrm6\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.006990 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.006889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a0275-09dc-4108-9557-6cf76e6117b5-metrics-client-ca\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.006990 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.006912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.006990 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.006945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b3f5c6a-d31b-415f-9166-af11041f6afc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.006990 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.006967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc2a0275-09dc-4108-9557-6cf76e6117b5-sys\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.007221 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.007003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b3f5c6a-d31b-415f-9166-af11041f6afc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.007221 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.007033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncq7j\" (UniqueName: \"kubernetes.io/projected/cc2a0275-09dc-4108-9557-6cf76e6117b5-kube-api-access-ncq7j\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.007221 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.007065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc2a0275-09dc-4108-9557-6cf76e6117b5-root\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.007221 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.007147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-textfile\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.007412 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.007246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-tls\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.007412 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.007273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.007412 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.007302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b3f5c6a-d31b-415f-9166-af11041f6afc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.007412 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.007333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-wtmp\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.108369 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-tls\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.108369 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.108580 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b3f5c6a-d31b-415f-9166-af11041f6afc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.108580 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-wtmp\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.108580 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrm6\" (UniqueName: \"kubernetes.io/projected/7b3f5c6a-d31b-415f-9166-af11041f6afc-kube-api-access-wmrm6\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.108580 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a0275-09dc-4108-9557-6cf76e6117b5-metrics-client-ca\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.108580 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.108580 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b3f5c6a-d31b-415f-9166-af11041f6afc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.108580 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc2a0275-09dc-4108-9557-6cf76e6117b5-sys\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.108886 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b3f5c6a-d31b-415f-9166-af11041f6afc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.108886 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncq7j\" (UniqueName: \"kubernetes.io/projected/cc2a0275-09dc-4108-9557-6cf76e6117b5-kube-api-access-ncq7j\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.108886 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc2a0275-09dc-4108-9557-6cf76e6117b5-root\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.108886 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.108705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-textfile\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.109131 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.109108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-textfile\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.109463 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:46:28.109444    2578 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 20:46:28.109560 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:46:28.109523    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b3f5c6a-d31b-415f-9166-af11041f6afc-openshift-state-metrics-tls podName:7b3f5c6a-d31b-415f-9166-af11041f6afc nodeName:}" failed. No retries permitted until 2026-07-28 20:46:28.609502952 +0000 UTC m=+77.464346851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7b3f5c6a-d31b-415f-9166-af11041f6afc-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-455lz" (UID: "7b3f5c6a-d31b-415f-9166-af11041f6afc") : secret "openshift-state-metrics-tls" not found
Jul 28 20:46:28.109637 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.109616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2a0275-09dc-4108-9557-6cf76e6117b5-metrics-client-ca\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.109707 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.109692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-wtmp\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.109830 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.109813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc2a0275-09dc-4108-9557-6cf76e6117b5-root\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.109906 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.109836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc2a0275-09dc-4108-9557-6cf76e6117b5-sys\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.110472 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.110441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.110630 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.110575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b3f5c6a-d31b-415f-9166-af11041f6afc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.112990 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.112962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-tls\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.114540 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.114519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc2a0275-09dc-4108-9557-6cf76e6117b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.115266 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.115240    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b3f5c6a-d31b-415f-9166-af11041f6afc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.119226 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.119205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncq7j\" (UniqueName: \"kubernetes.io/projected/cc2a0275-09dc-4108-9557-6cf76e6117b5-kube-api-access-ncq7j\") pod \"node-exporter-6xrrf\" (UID: \"cc2a0275-09dc-4108-9557-6cf76e6117b5\") " pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.121662 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.121637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrm6\" (UniqueName: \"kubernetes.io/projected/7b3f5c6a-d31b-415f-9166-af11041f6afc-kube-api-access-wmrm6\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.299900 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.299865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6xrrf"
Jul 28 20:46:28.613801 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.613726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b3f5c6a-d31b-415f-9166-af11041f6afc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.616599 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.616572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b3f5c6a-d31b-415f-9166-af11041f6afc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-455lz\" (UID: \"7b3f5c6a-d31b-415f-9166-af11041f6afc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:28.864636 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:28.864549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"
Jul 28 20:46:29.015215 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.013486    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 20:46:29.018851 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.018826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.021382 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.021362    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 20:46:29.021530 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.021394    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 20:46:29.023441 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.023423    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 20:46:29.023834 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.023814    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 20:46:29.024396 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.024059    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 20:46:29.024396 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.024095    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 20:46:29.024396 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.024197    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 20:46:29.024396 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.024236    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 20:46:29.024396 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.024342    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 20:46:29.024865 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.024842    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n9sc4\""
Jul 28 20:46:29.032479 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.032458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 20:46:29.117047 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.116955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-config-out\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117047 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-web-config\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117047 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjcl4\" (UniqueName: \"kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-kube-api-access-vjcl4\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.117635 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.117354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218458 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218635 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218635 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218635 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-config-out\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218635 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-web-config\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218635 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218635 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218962 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218962 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218962 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjcl4\" (UniqueName: \"kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-kube-api-access-vjcl4\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218962 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218962 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.218962 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.218797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.219826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.219794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.220870 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.219963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.220870 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.220802    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.222389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.222356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.222492 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.222418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.224171 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.223430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-web-config\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.224171 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.223694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.224171 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.223952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.224171 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.224136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.224610 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.224587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.224695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.224589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-config-out\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.224864 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.224844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.228228 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.228205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjcl4\" (UniqueName: \"kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-kube-api-access-vjcl4\") pod \"alertmanager-main-0\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:29.329870 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:29.329830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:46:30.880712 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:30.880667    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2a0275_09dc_4108_9557_6cf76e6117b5.slice/crio-17f34285f46444e4344ff14097a6b1a320a7f766e4d113abb0e58cdaac897841 WatchSource:0}: Error finding container 17f34285f46444e4344ff14097a6b1a320a7f766e4d113abb0e58cdaac897841: Status 404 returned error can't find the container with id 17f34285f46444e4344ff14097a6b1a320a7f766e4d113abb0e58cdaac897841
Jul 28 20:46:31.050108 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.050019    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz"]
Jul 28 20:46:31.052328 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:31.052297    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3f5c6a_d31b_415f_9166_af11041f6afc.slice/crio-f777827c745886800922c0e00c606ea8d1155bdf00eb5ac1cdc4576a6fbad1ef WatchSource:0}: Error finding container f777827c745886800922c0e00c606ea8d1155bdf00eb5ac1cdc4576a6fbad1ef: Status 404 returned error can't find the container with id f777827c745886800922c0e00c606ea8d1155bdf00eb5ac1cdc4576a6fbad1ef
Jul 28 20:46:31.069506 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.069061    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 20:46:31.073392 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:31.073364    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10340074_7474_46f2_be16_d4defe28a2d3.slice/crio-2bbdaad99b436c38d84c3f4a2bbb9b2016a45d35f4777bad9e575f01a1ddd791 WatchSource:0}: Error finding container 2bbdaad99b436c38d84c3f4a2bbb9b2016a45d35f4777bad9e575f01a1ddd791: Status 404 returned error can't find the container with id 2bbdaad99b436c38d84c3f4a2bbb9b2016a45d35f4777bad9e575f01a1ddd791
Jul 28 20:46:31.889255 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.889210    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-vr4rz" event={"ID":"98b94339-faa6-414e-be68-c6ec91e9b4eb","Type":"ContainerStarted","Data":"2f133e6ce0498278571c3bf521c5cae8e711d07e6e845e28f3b2cabe8881ad59"}
Jul 28 20:46:31.889825 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.889799    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-vr4rz"
Jul 28 20:46:31.892150 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.892033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xrrf" event={"ID":"cc2a0275-09dc-4108-9557-6cf76e6117b5","Type":"ContainerStarted","Data":"17f34285f46444e4344ff14097a6b1a320a7f766e4d113abb0e58cdaac897841"}
Jul 28 20:46:31.895318 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.895290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz" event={"ID":"7b3f5c6a-d31b-415f-9166-af11041f6afc","Type":"ContainerStarted","Data":"6104aabcafc2950b1ea4f2a6d16c50ab3141670e3f8ccda5744bab1f4a80fede"}
Jul 28 20:46:31.895428 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.895344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz" event={"ID":"7b3f5c6a-d31b-415f-9166-af11041f6afc","Type":"ContainerStarted","Data":"f22c0bff8377c3b74e73fcd1f4bc5d518db40289014873c42a6b4c8987c08327"}
Jul 28 20:46:31.895428 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.895373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz" event={"ID":"7b3f5c6a-d31b-415f-9166-af11041f6afc","Type":"ContainerStarted","Data":"f777827c745886800922c0e00c606ea8d1155bdf00eb5ac1cdc4576a6fbad1ef"}
Jul 28 20:46:31.897463 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.897378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerStarted","Data":"2bbdaad99b436c38d84c3f4a2bbb9b2016a45d35f4777bad9e575f01a1ddd791"}
Jul 28 20:46:31.902876 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.902853    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-vr4rz"
Jul 28 20:46:31.913103 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:31.912221    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-vr4rz" podStartSLOduration=2.303677415 podStartE2EDuration="19.912203453s" totalImagesPullingTime="17.608526038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:31.910949024 +0000 UTC m=+80.765792942" watchObservedRunningTime="2026-07-28 20:46:31.912203453 +0000 UTC m=+80.767047365"
Jul 28 20:46:32.338424 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.337119    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-855b84fc64-nmz64"]
Jul 28 20:46:32.359706 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.357400    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-855b84fc64-nmz64"]
Jul 28 20:46:32.359706 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.357527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.371373 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.360537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 20:46:32.371373 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.360772    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 20:46:32.371373 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.360993    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hrpkd\""
Jul 28 20:46:32.374649 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.372721    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 20:46:32.374649 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.373020    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2c3tcrpur3pfd\""
Jul 28 20:46:32.374649 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.373259    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 20:46:32.455100 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.455036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5d72cae-f158-41eb-9d7e-60f5633d7466-secret-metrics-server-client-certs\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.455267 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.455138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d72cae-f158-41eb-9d7e-60f5633d7466-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.455267 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.455195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5d72cae-f158-41eb-9d7e-60f5633d7466-metrics-server-audit-profiles\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.455267 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.455234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5d72cae-f158-41eb-9d7e-60f5633d7466-secret-metrics-server-tls\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.455438 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.455266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5d72cae-f158-41eb-9d7e-60f5633d7466-client-ca-bundle\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.455438 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.455299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mpks\" (UniqueName: \"kubernetes.io/projected/e5d72cae-f158-41eb-9d7e-60f5633d7466-kube-api-access-5mpks\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.455438 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.455353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5d72cae-f158-41eb-9d7e-60f5633d7466-audit-log\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.557389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.556941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5d72cae-f158-41eb-9d7e-60f5633d7466-audit-log\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.557389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.557001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5d72cae-f158-41eb-9d7e-60f5633d7466-secret-metrics-server-client-certs\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.557389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.557051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d72cae-f158-41eb-9d7e-60f5633d7466-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.557389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.557120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5d72cae-f158-41eb-9d7e-60f5633d7466-metrics-server-audit-profiles\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.557389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.557154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5d72cae-f158-41eb-9d7e-60f5633d7466-secret-metrics-server-tls\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.557389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.557182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5d72cae-f158-41eb-9d7e-60f5633d7466-client-ca-bundle\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.557389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.557214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpks\" (UniqueName: \"kubernetes.io/projected/e5d72cae-f158-41eb-9d7e-60f5633d7466-kube-api-access-5mpks\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.557872 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.557728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5d72cae-f158-41eb-9d7e-60f5633d7466-audit-log\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.558849 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.558809    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d72cae-f158-41eb-9d7e-60f5633d7466-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.558980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.558866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5d72cae-f158-41eb-9d7e-60f5633d7466-metrics-server-audit-profiles\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.561211 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.561164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5d72cae-f158-41eb-9d7e-60f5633d7466-client-ca-bundle\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.561476 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.561454    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5d72cae-f158-41eb-9d7e-60f5633d7466-secret-metrics-server-tls\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.563526 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.563487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5d72cae-f158-41eb-9d7e-60f5633d7466-secret-metrics-server-client-certs\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.569491 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.569468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpks\" (UniqueName: \"kubernetes.io/projected/e5d72cae-f158-41eb-9d7e-60f5633d7466-kube-api-access-5mpks\") pod \"metrics-server-855b84fc64-nmz64\" (UID: \"e5d72cae-f158-41eb-9d7e-60f5633d7466\") " pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:32.688781 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:32.688442    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:34.287345 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.287288    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 20:46:34.302665 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.302638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.306721 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.306698    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d9nc223vlqt2u\""
Jul 28 20:46:34.306850 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.306702    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 20:46:34.307521 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.307466    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 20:46:34.308011 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.307665    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t2gr7\""
Jul 28 20:46:34.309566 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.309541    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 20:46:34.309978 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.309956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 20:46:34.311385 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.310498    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 20:46:34.311385 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.310528    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 20:46:34.312311 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.311889    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 20:46:34.312311 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.312214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 20:46:34.313282 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.312607    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 20:46:34.313282 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.312803    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 20:46:34.313282 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.312999    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 20:46:34.313725 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.313630    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 20:46:34.315259 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.314281    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 20:46:34.315259 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.314518    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 20:46:34.321678 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.321638    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config-out\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzv8m\" (UniqueName: \"kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-kube-api-access-wzv8m\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-web-config\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.375817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.376611 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.376611 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.376611 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.376611 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.375643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.476911 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.476863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477124 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.476922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config-out\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477124 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.476959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477124 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.476984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477124 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477124 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477124 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzv8m\" (UniqueName: \"kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-kube-api-access-wzv8m\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-web-config\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.477475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.478019 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.477503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.478201 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.478180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.480346 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.479637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.482595 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.482289    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.483832 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.483015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.483832 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.483390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.483832 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.483714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.484574 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.484186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.484988 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.484621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.484988 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.484919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.485768 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.485699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.486128 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.486070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.486128 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.486093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config-out\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.486613 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.486589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.489013 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.488983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.490167 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.490146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.490903 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.490682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.490903 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.490783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.491381 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.491181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.491381 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.491344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-web-config\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.492303 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.492268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzv8m\" (UniqueName: \"kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-kube-api-access-wzv8m\") pod \"prometheus-k8s-0\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.618689 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.618599    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:34.813178 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:34.813067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65c598f646-wbnmg"
Jul 28 20:46:38.315738 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.315715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-855b84fc64-nmz64"]
Jul 28 20:46:38.320860 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:38.320726    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d72cae_f158_41eb_9d7e_60f5633d7466.slice/crio-fd3ac1b105b150f4a229acc1ed6dec4eea3dae1a2157ab33316c3fc6a3c54b3e WatchSource:0}: Error finding container fd3ac1b105b150f4a229acc1ed6dec4eea3dae1a2157ab33316c3fc6a3c54b3e: Status 404 returned error can't find the container with id fd3ac1b105b150f4a229acc1ed6dec4eea3dae1a2157ab33316c3fc6a3c54b3e
Jul 28 20:46:38.352121 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.352070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 20:46:38.357185 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:46:38.357157    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb6532b_c6e6_4e05_8f07_272b3daa28e5.slice/crio-91799d89266c3a22c4abf10da91fc756f11beffcbcf820c421fa658109805f08 WatchSource:0}: Error finding container 91799d89266c3a22c4abf10da91fc756f11beffcbcf820c421fa658109805f08: Status 404 returned error can't find the container with id 91799d89266c3a22c4abf10da91fc756f11beffcbcf820c421fa658109805f08
Jul 28 20:46:38.929046 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.928955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-855b84fc64-nmz64" event={"ID":"e5d72cae-f158-41eb-9d7e-60f5633d7466","Type":"ContainerStarted","Data":"fd3ac1b105b150f4a229acc1ed6dec4eea3dae1a2157ab33316c3fc6a3c54b3e"}
Jul 28 20:46:38.931008 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.930981    2578 generic.go:356] "Generic (PLEG): container finished" podID="cc2a0275-09dc-4108-9557-6cf76e6117b5" containerID="66c10f7eb3ff2b66c6759fc50e51c259c26967bd6e24f3ae4ec6818ad8f12cd4" exitCode=0
Jul 28 20:46:38.931184 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.931018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xrrf" event={"ID":"cc2a0275-09dc-4108-9557-6cf76e6117b5","Type":"ContainerDied","Data":"66c10f7eb3ff2b66c6759fc50e51c259c26967bd6e24f3ae4ec6818ad8f12cd4"}
Jul 28 20:46:38.933033 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.933006    2578 generic.go:356] "Generic (PLEG): container finished" podID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerID="d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f" exitCode=0
Jul 28 20:46:38.933177 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.933102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerDied","Data":"d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f"}
Jul 28 20:46:38.933244 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.933177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerStarted","Data":"91799d89266c3a22c4abf10da91fc756f11beffcbcf820c421fa658109805f08"}
Jul 28 20:46:38.936860 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.936258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz" event={"ID":"7b3f5c6a-d31b-415f-9166-af11041f6afc","Type":"ContainerStarted","Data":"a77ebaf99a4b7ae87a04ea89ea04c144a21b42985b9db40ffdc6cb48392f09f0"}
Jul 28 20:46:38.938068 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.938043    2578 generic.go:356] "Generic (PLEG): container finished" podID="10340074-7474-46f2-be16-d4defe28a2d3" containerID="d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6" exitCode=0
Jul 28 20:46:38.938210 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.938111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerDied","Data":"d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6"}
Jul 28 20:46:38.940064 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:38.940040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kg2x9" event={"ID":"349dc143-219c-4d79-a6a3-24778d73a200","Type":"ContainerStarted","Data":"f76033385160f8c389014b471f2b81c92fec224af89c49238dbdb1ad95bf1b4f"}
Jul 28 20:46:39.038659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:39.038605    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-455lz" podStartSLOduration=5.648253145 podStartE2EDuration="12.038590851s" totalImagesPullingTime="6.390337706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:39.037712336 +0000 UTC m=+87.892556255" watchObservedRunningTime="2026-07-28 20:46:39.038590851 +0000 UTC m=+87.893434761"
Jul 28 20:46:39.953972 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:39.953847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xrrf" event={"ID":"cc2a0275-09dc-4108-9557-6cf76e6117b5","Type":"ContainerStarted","Data":"a8b9df6386d460b3a898719e74e823a27ca92d6406ad2e875166509eca25b6eb"}
Jul 28 20:46:39.953972 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:39.953940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xrrf" event={"ID":"cc2a0275-09dc-4108-9557-6cf76e6117b5","Type":"ContainerStarted","Data":"4a809cc8e0a330f4ba8c59d8dc94ff40b8b1d12109a2f74f8a026d49d2f83c2d"}
Jul 28 20:46:39.981113 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:39.980833    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6xrrf" podStartSLOduration=5.644961851 podStartE2EDuration="12.980811836s" totalImagesPullingTime="7.236709821s" totalInitContainerRuntime="99.140164ms" isStatefulPod=true podCreationTimestamp="2026-07-28 20:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:39.979365959 +0000 UTC m=+88.834209872" watchObservedRunningTime="2026-07-28 20:46:39.980811836 +0000 UTC m=+88.835655749"
Jul 28 20:46:45.978506 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:45.978463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kg2x9" event={"ID":"349dc143-219c-4d79-a6a3-24778d73a200","Type":"ContainerStarted","Data":"1cbff70ed9a230a6096d36bdc69914819a22a8267619cbb53764312982f6713f"}
Jul 28 20:46:45.980098 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:45.980040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-855b84fc64-nmz64" event={"ID":"e5d72cae-f158-41eb-9d7e-60f5633d7466","Type":"ContainerStarted","Data":"341d819af4d24b812ce68cac9b8ff5d9d580044b806418680863b81c28739928"}
Jul 28 20:46:45.982404 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:45.982377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerStarted","Data":"46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac"}
Jul 28 20:46:45.982502 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:45.982461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerStarted","Data":"6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4"}
Jul 28 20:46:45.985157 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:45.985133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerStarted","Data":"3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c"}
Jul 28 20:46:45.985255 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:45.985203    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerStarted","Data":"1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d"}
Jul 28 20:46:45.985255 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:45.985246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerStarted","Data":"bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895"}
Jul 28 20:46:46.001916 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:46.001877    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-855b84fc64-nmz64" podStartSLOduration=6.915907433 podStartE2EDuration="14.00186368s" totalImagesPullingTime="7.085956247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:46.001172812 +0000 UTC m=+94.856016746" watchObservedRunningTime="2026-07-28 20:46:46.00186368 +0000 UTC m=+94.856707588"
Jul 28 20:46:46.994448 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:46.994286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerStarted","Data":"a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597"}
Jul 28 20:46:46.994448 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:46.994414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerStarted","Data":"a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83"}
Jul 28 20:46:51.012047 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:51.012017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerStarted","Data":"9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b"}
Jul 28 20:46:51.012444 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:51.012127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerStarted","Data":"eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2"}
Jul 28 20:46:51.012444 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:51.012165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerStarted","Data":"af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65"}
Jul 28 20:46:51.012444 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:51.012197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerStarted","Data":"ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939"}
Jul 28 20:46:51.015186 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:51.015156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerStarted","Data":"c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468"}
Jul 28 20:46:51.017098 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:51.017055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kg2x9" event={"ID":"349dc143-219c-4d79-a6a3-24778d73a200","Type":"ContainerStarted","Data":"1e989a69f76cd2808cdd15f31db490a27d6468f7bad8fceaa013b0c3e4ae712b"}
Jul 28 20:46:51.053672 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:51.053621    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.765189941 podStartE2EDuration="17.053605311s" totalImagesPullingTime="11.263019072s" totalInitContainerRuntime="25.396298ms" isStatefulPod=false podCreationTimestamp="2026-07-28 20:46:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:51.051125094 +0000 UTC m=+99.905969007" watchObservedRunningTime="2026-07-28 20:46:51.053605311 +0000 UTC m=+99.908449221"
Jul 28 20:46:51.075995 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:51.075911    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kg2x9" podStartSLOduration=36.27363244 podStartE2EDuration="1m3.075896819s" totalImagesPullingTime="26.802264379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:45:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:51.074678451 +0000 UTC m=+99.929522358" watchObservedRunningTime="2026-07-28 20:46:51.075896819 +0000 UTC m=+99.930740730"
Jul 28 20:46:51.108757 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:51.108708    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.705812315 podStartE2EDuration="23.108689326s" totalImagesPullingTime="17.3799571s" totalInitContainerRuntime="22.919911ms" isStatefulPod=false podCreationTimestamp="2026-07-28 20:46:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:46:51.107161363 +0000 UTC m=+99.962005277" watchObservedRunningTime="2026-07-28 20:46:51.108689326 +0000 UTC m=+99.963533237"
Jul 28 20:46:52.689035 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:52.689001    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:52.689501 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:52.689106    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:46:54.619139 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:54.619105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:46:54.860427 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:46:54.860398    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tsl56"
Jul 28 20:47:12.694766 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:12.694736    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:47:12.698494 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:12.698474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-855b84fc64-nmz64"
Jul 28 20:47:34.619146 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:34.619109    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:34.637968 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:34.637945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:35.197556 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:35.197531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:48.549589 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:48.549552    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 20:47:48.550194 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:48.549820    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="alertmanager" containerID="cri-o://bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895" gracePeriod=120
Jul 28 20:47:48.550194 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:48.549881    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy-metric" containerID="cri-o://a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597" gracePeriod=120
Jul 28 20:47:48.550194 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:48.549913    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="prom-label-proxy" containerID="cri-o://c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468" gracePeriod=120
Jul 28 20:47:48.550194 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:48.549956    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy-web" containerID="cri-o://3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c" gracePeriod=120
Jul 28 20:47:48.550194 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:48.549971    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy" containerID="cri-o://a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83" gracePeriod=120
Jul 28 20:47:48.550451 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:48.550394    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="config-reloader" containerID="cri-o://1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d" gracePeriod=120
Jul 28 20:47:49.235046 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.235013    2578 generic.go:356] "Generic (PLEG): container finished" podID="10340074-7474-46f2-be16-d4defe28a2d3" containerID="c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468" exitCode=0
Jul 28 20:47:49.235046 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.235037    2578 generic.go:356] "Generic (PLEG): container finished" podID="10340074-7474-46f2-be16-d4defe28a2d3" containerID="a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597" exitCode=0
Jul 28 20:47:49.235046 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.235043    2578 generic.go:356] "Generic (PLEG): container finished" podID="10340074-7474-46f2-be16-d4defe28a2d3" containerID="a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83" exitCode=0
Jul 28 20:47:49.235046 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.235049    2578 generic.go:356] "Generic (PLEG): container finished" podID="10340074-7474-46f2-be16-d4defe28a2d3" containerID="1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d" exitCode=0
Jul 28 20:47:49.235046 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.235054    2578 generic.go:356] "Generic (PLEG): container finished" podID="10340074-7474-46f2-be16-d4defe28a2d3" containerID="bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895" exitCode=0
Jul 28 20:47:49.235337 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.235101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerDied","Data":"c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468"}
Jul 28 20:47:49.235337 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.235187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerDied","Data":"a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597"}
Jul 28 20:47:49.235337 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.235218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerDied","Data":"a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83"}
Jul 28 20:47:49.235337 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.235241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerDied","Data":"1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d"}
Jul 28 20:47:49.235337 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.235264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerDied","Data":"bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895"}
Jul 28 20:47:49.799268 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.799246    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:47:49.826783 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.826757    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-main-tls\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.826905 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.826798    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-main-db\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.826905 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.826827    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.826905 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.826873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.826905 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.826903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-config-volume\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.827152 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.826995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjcl4\" (UniqueName: \"kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-kube-api-access-vjcl4\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.827152 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.827041    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-tls-assets\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.827152 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.827093    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-config-out\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.827152 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.827122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-trusted-ca-bundle\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.827370 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.827147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:47:49.827370 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.827173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-cluster-tls-config\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.827370 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.827212    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-web-config\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.827370 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.827240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-metrics-client-ca\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.827370 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.827272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"10340074-7474-46f2-be16-d4defe28a2d3\" (UID: \"10340074-7474-46f2-be16-d4defe28a2d3\") "
Jul 28 20:47:49.827633 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.827495    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-main-db\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.828634 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.828594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:49.828746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.828638    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:49.829898 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.829874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-config-out" (OuterVolumeSpecName: "config-out") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:47:49.830272 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.830248    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:47:49.830375 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.830267    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-config-volume" (OuterVolumeSpecName: "config-volume") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:49.830375 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.830283    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:49.830495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.830266    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:49.830495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.830412    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:49.830495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.830464    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:49.831586 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.831543    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-kube-api-access-vjcl4" (OuterVolumeSpecName: "kube-api-access-vjcl4") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "kube-api-access-vjcl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:47:49.837517 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.837482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:49.843465 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.843442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-web-config" (OuterVolumeSpecName: "web-config") pod "10340074-7474-46f2-be16-d4defe28a2d3" (UID: "10340074-7474-46f2-be16-d4defe28a2d3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:49.928327 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928261    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-main-tls\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928327 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928286    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928327 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928296    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928327 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928305    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928327 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928313    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjcl4\" (UniqueName: \"kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-kube-api-access-vjcl4\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928327 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928321    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10340074-7474-46f2-be16-d4defe28a2d3-tls-assets\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928327 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928329    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10340074-7474-46f2-be16-d4defe28a2d3-config-out\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928609 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928337    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928609 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928345    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-cluster-tls-config\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928609 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928353    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-web-config\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928609 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928360    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10340074-7474-46f2-be16-d4defe28a2d3-metrics-client-ca\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:49.928609 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:49.928368    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/10340074-7474-46f2-be16-d4defe28a2d3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:50.241921 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.241889    2578 generic.go:356] "Generic (PLEG): container finished" podID="10340074-7474-46f2-be16-d4defe28a2d3" containerID="3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c" exitCode=0
Jul 28 20:47:50.242060 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.241967    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 20:47:50.242060 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.241972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerDied","Data":"3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c"}
Jul 28 20:47:50.242060 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.242040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"10340074-7474-46f2-be16-d4defe28a2d3","Type":"ContainerDied","Data":"2bbdaad99b436c38d84c3f4a2bbb9b2016a45d35f4777bad9e575f01a1ddd791"}
Jul 28 20:47:50.242183 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.242087    2578 scope.go:117] "RemoveContainer" containerID="c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468"
Jul 28 20:47:50.250287 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.250214    2578 scope.go:117] "RemoveContainer" containerID="a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597"
Jul 28 20:47:50.258534 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.258519    2578 scope.go:117] "RemoveContainer" containerID="a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83"
Jul 28 20:47:50.264960 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.264945    2578 scope.go:117] "RemoveContainer" containerID="3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c"
Jul 28 20:47:50.269377 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.269343    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 20:47:50.271775 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.271758    2578 scope.go:117] "RemoveContainer" containerID="1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d"
Jul 28 20:47:50.273954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.273932    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 20:47:50.278985 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.278962    2578 scope.go:117] "RemoveContainer" containerID="bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895"
Jul 28 20:47:50.285538 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.285523    2578 scope.go:117] "RemoveContainer" containerID="d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6"
Jul 28 20:47:50.291946 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.291931    2578 scope.go:117] "RemoveContainer" containerID="c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468"
Jul 28 20:47:50.292221 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:50.292203    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468\": container with ID starting with c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468 not found: ID does not exist" containerID="c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468"
Jul 28 20:47:50.292277 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.292230    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468"} err="failed to get container status \"c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468\": rpc error: code = NotFound desc = could not find container \"c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468\": container with ID starting with c0730a2e35b6c856f2f05b827a6e8d55bb4070836b3353df6a7da74925c18468 not found: ID does not exist"
Jul 28 20:47:50.292277 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.292257    2578 scope.go:117] "RemoveContainer" containerID="a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597"
Jul 28 20:47:50.292513 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:50.292498    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597\": container with ID starting with a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597 not found: ID does not exist" containerID="a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597"
Jul 28 20:47:50.292618 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.292519    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597"} err="failed to get container status \"a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597\": rpc error: code = NotFound desc = could not find container \"a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597\": container with ID starting with a7fb9a30ad4f07bb44c94b93e906b6cba97da916db5cc279da3e3fcaaadd6597 not found: ID does not exist"
Jul 28 20:47:50.292618 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.292530    2578 scope.go:117] "RemoveContainer" containerID="a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83"
Jul 28 20:47:50.292733 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:50.292717    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83\": container with ID starting with a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83 not found: ID does not exist" containerID="a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83"
Jul 28 20:47:50.292771 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.292736    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83"} err="failed to get container status \"a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83\": rpc error: code = NotFound desc = could not find container \"a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83\": container with ID starting with a42b02935565141a610390cf13d97095d81f21fa5d32bd1a6c537dd94413ac83 not found: ID does not exist"
Jul 28 20:47:50.292771 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.292748    2578 scope.go:117] "RemoveContainer" containerID="3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c"
Jul 28 20:47:50.292900 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:50.292886    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c\": container with ID starting with 3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c not found: ID does not exist" containerID="3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c"
Jul 28 20:47:50.292939 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.292904    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c"} err="failed to get container status \"3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c\": rpc error: code = NotFound desc = could not find container \"3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c\": container with ID starting with 3a257c77089be795425d9611e01035d86897578638baadcef06dc330b7d0963c not found: ID does not exist"
Jul 28 20:47:50.292939 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.292917    2578 scope.go:117] "RemoveContainer" containerID="1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d"
Jul 28 20:47:50.293132 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:50.293064    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d\": container with ID starting with 1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d not found: ID does not exist" containerID="1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d"
Jul 28 20:47:50.293132 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.293098    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d"} err="failed to get container status \"1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d\": rpc error: code = NotFound desc = could not find container \"1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d\": container with ID starting with 1701570fe591bd81668be967e5753d28f4402e00ce6705fa2fd1ea1f3687862d not found: ID does not exist"
Jul 28 20:47:50.293132 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.293117    2578 scope.go:117] "RemoveContainer" containerID="bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895"
Jul 28 20:47:50.293350 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:50.293335    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895\": container with ID starting with bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895 not found: ID does not exist" containerID="bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895"
Jul 28 20:47:50.293405 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.293353    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895"} err="failed to get container status \"bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895\": rpc error: code = NotFound desc = could not find container \"bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895\": container with ID starting with bcb500e6519610db273c1cf245fcd38e32f6d7ddf40c186826efd9d3ababc895 not found: ID does not exist"
Jul 28 20:47:50.293405 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.293364    2578 scope.go:117] "RemoveContainer" containerID="d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6"
Jul 28 20:47:50.293607 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:50.293590    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6\": container with ID starting with d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6 not found: ID does not exist" containerID="d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6"
Jul 28 20:47:50.293668 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:50.293615    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6"} err="failed to get container status \"d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6\": rpc error: code = NotFound desc = could not find container \"d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6\": container with ID starting with d589f930377b8569f89f1e7e3a5ef759d28a386c1163807703359d966db93bc6 not found: ID does not exist"
Jul 28 20:47:51.527968 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:51.527932    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10340074-7474-46f2-be16-d4defe28a2d3" path="/var/lib/kubelet/pods/10340074-7474-46f2-be16-d4defe28a2d3/volumes"
Jul 28 20:47:52.771063 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:52.771033    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 20:47:52.771471 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:52.771305    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="prometheus" containerID="cri-o://6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4" gracePeriod=600
Jul 28 20:47:52.771471 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:52.771348    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy" containerID="cri-o://eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2" gracePeriod=600
Jul 28 20:47:52.771471 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:52.771384    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy-web" containerID="cri-o://af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65" gracePeriod=600
Jul 28 20:47:52.771471 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:52.771399    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b" gracePeriod=600
Jul 28 20:47:52.771679 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:52.771392    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="thanos-sidecar" containerID="cri-o://ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939" gracePeriod=600
Jul 28 20:47:52.771679 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:52.771429    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="config-reloader" containerID="cri-o://46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac" gracePeriod=600
Jul 28 20:47:53.255976 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:53.255945    2578 generic.go:356] "Generic (PLEG): container finished" podID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerID="9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b" exitCode=0
Jul 28 20:47:53.255976 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:53.255968    2578 generic.go:356] "Generic (PLEG): container finished" podID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerID="eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2" exitCode=0
Jul 28 20:47:53.255976 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:53.255975    2578 generic.go:356] "Generic (PLEG): container finished" podID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerID="ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939" exitCode=0
Jul 28 20:47:53.255976 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:53.255981    2578 generic.go:356] "Generic (PLEG): container finished" podID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerID="46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac" exitCode=0
Jul 28 20:47:53.255976 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:53.255986    2578 generic.go:356] "Generic (PLEG): container finished" podID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerID="6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4" exitCode=0
Jul 28 20:47:53.256285 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:53.256020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerDied","Data":"9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b"}
Jul 28 20:47:53.256285 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:53.256100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerDied","Data":"eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2"}
Jul 28 20:47:53.256285 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:53.256128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerDied","Data":"ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939"}
Jul 28 20:47:53.256285 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:53.256155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerDied","Data":"46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac"}
Jul 28 20:47:53.256285 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:53.256178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerDied","Data":"6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4"}
Jul 28 20:47:54.022794 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.022772    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.056570 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-0\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056699 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-metrics-client-ca\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056699 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056795 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056720    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-serving-certs-ca-bundle\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056795 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-trusted-ca-bundle\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056795 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config-out\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056795 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056792    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-1\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056984 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056816    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-db\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056984 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056847    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-web-config\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056984 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056876    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzv8m\" (UniqueName: \"kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-kube-api-access-wzv8m\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056984 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-kubelet-serving-ca-bundle\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056984 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056930    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-thanos-prometheus-http-client-file\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.056984 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.057300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.056996    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.057300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057018    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-grpc-tls\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.057300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057038    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:54.057300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-2\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.057300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057109    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:54.057300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057128    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-tls\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.057300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057237    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-tls-assets\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.057300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-metrics-client-certs\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.057676 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-kube-rbac-proxy\") pod \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\" (UID: \"bfb6532b-c6e6-4e05-8f07-272b3daa28e5\") "
Jul 28 20:47:54.057676 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057480    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:54.057676 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057524    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:54.057896 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057875    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-metrics-client-ca\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.057965 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057904    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.057965 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057921    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.057965 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057937    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.058118 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.057353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:54.058226 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.058179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:47:54.059248 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.058930    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:54.060444 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.060386    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config-out" (OuterVolumeSpecName: "config-out") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:47:54.061768 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.061734    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:54.061866 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.061762    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:54.062354 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.062315    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:47:54.062531 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.062503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:47:54.062660 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.062636    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:54.063173 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.063141    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config" (OuterVolumeSpecName: "config") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:54.063389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.063349    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:54.063389 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.063362    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:54.064115 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.064052    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:54.064351 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.064321    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-kube-api-access-wzv8m" (OuterVolumeSpecName: "kube-api-access-wzv8m") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "kube-api-access-wzv8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:47:54.064460 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.064425    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:54.074063 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.074039    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-web-config" (OuterVolumeSpecName: "web-config") pod "bfb6532b-c6e6-4e05-8f07-272b3daa28e5" (UID: "bfb6532b-c6e6-4e05-8f07-272b3daa28e5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:47:54.159040 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.158955    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-web-config\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159040 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.158997    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzv8m\" (UniqueName: \"kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-kube-api-access-wzv8m\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159040 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159007    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159040 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159018    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159040 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159027    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159040 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159036    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159040 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159044    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-grpc-tls\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159040 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159053    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159390 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159060    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159390 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159068    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-tls-assets\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159390 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159093    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-metrics-client-certs\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159390 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159103    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-kube-rbac-proxy\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159390 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159113    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159390 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159121    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159390 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159131    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-config-out\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.159390 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.159140    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfb6532b-c6e6-4e05-8f07-272b3daa28e5-prometheus-k8s-db\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:47:54.263632 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.263600    2578 generic.go:356] "Generic (PLEG): container finished" podID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerID="af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65" exitCode=0
Jul 28 20:47:54.263786 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.263671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerDied","Data":"af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65"}
Jul 28 20:47:54.263786 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.263682    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.263786 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.263729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfb6532b-c6e6-4e05-8f07-272b3daa28e5","Type":"ContainerDied","Data":"91799d89266c3a22c4abf10da91fc756f11beffcbcf820c421fa658109805f08"}
Jul 28 20:47:54.263786 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.263751    2578 scope.go:117] "RemoveContainer" containerID="9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b"
Jul 28 20:47:54.271316 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.271300    2578 scope.go:117] "RemoveContainer" containerID="eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2"
Jul 28 20:47:54.278198 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.278181    2578 scope.go:117] "RemoveContainer" containerID="af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65"
Jul 28 20:47:54.284468 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.284453    2578 scope.go:117] "RemoveContainer" containerID="ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939"
Jul 28 20:47:54.289183 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.289164    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 20:47:54.290832 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.290817    2578 scope.go:117] "RemoveContainer" containerID="46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac"
Jul 28 20:47:54.294989 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.294961    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 20:47:54.297577 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.297561    2578 scope.go:117] "RemoveContainer" containerID="6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4"
Jul 28 20:47:54.304171 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.304157    2578 scope.go:117] "RemoveContainer" containerID="d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f"
Jul 28 20:47:54.310193 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.310128    2578 scope.go:117] "RemoveContainer" containerID="9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b"
Jul 28 20:47:54.310382 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:54.310365    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b\": container with ID starting with 9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b not found: ID does not exist" containerID="9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b"
Jul 28 20:47:54.310435 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.310389    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b"} err="failed to get container status \"9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b\": rpc error: code = NotFound desc = could not find container \"9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b\": container with ID starting with 9c7f82a73ac119fb3f0fab01874c66ca8872f13f045fea74fd701ae623ac467b not found: ID does not exist"
Jul 28 20:47:54.310435 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.310408    2578 scope.go:117] "RemoveContainer" containerID="eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2"
Jul 28 20:47:54.310653 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:54.310635    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2\": container with ID starting with eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2 not found: ID does not exist" containerID="eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2"
Jul 28 20:47:54.314185 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.310662    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2"} err="failed to get container status \"eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2\": rpc error: code = NotFound desc = could not find container \"eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2\": container with ID starting with eaf5ff1f9b0840a1d4f686a6dfbb0e73955b64222222634ba62b719c08ed28d2 not found: ID does not exist"
Jul 28 20:47:54.314185 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.310981    2578 scope.go:117] "RemoveContainer" containerID="af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65"
Jul 28 20:47:54.315404 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:54.315374    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65\": container with ID starting with af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65 not found: ID does not exist" containerID="af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65"
Jul 28 20:47:54.315495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.315412    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65"} err="failed to get container status \"af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65\": rpc error: code = NotFound desc = could not find container \"af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65\": container with ID starting with af4b49fdaef0e173f7a9c1f73d9930e0ffe3b4562f6fcd474bc93346a26bcd65 not found: ID does not exist"
Jul 28 20:47:54.315495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.315435    2578 scope.go:117] "RemoveContainer" containerID="ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939"
Jul 28 20:47:54.315695 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:54.315671    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939\": container with ID starting with ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939 not found: ID does not exist" containerID="ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939"
Jul 28 20:47:54.315750 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.315701    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939"} err="failed to get container status \"ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939\": rpc error: code = NotFound desc = could not find container \"ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939\": container with ID starting with ff9c6a9c4f0028969aa02afd91bf040ebcf03c42ecec06218518383d85bd4939 not found: ID does not exist"
Jul 28 20:47:54.315750 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.315721    2578 scope.go:117] "RemoveContainer" containerID="46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac"
Jul 28 20:47:54.315980 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:54.315958    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac\": container with ID starting with 46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac not found: ID does not exist" containerID="46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac"
Jul 28 20:47:54.316035 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.315985    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac"} err="failed to get container status \"46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac\": rpc error: code = NotFound desc = could not find container \"46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac\": container with ID starting with 46aa75de1881d4b2bbbca2f143ee00c3fa7c10747d4a4b89cdeba81e34159aac not found: ID does not exist"
Jul 28 20:47:54.316035 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.316001    2578 scope.go:117] "RemoveContainer" containerID="6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4"
Jul 28 20:47:54.316242 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:54.316224    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4\": container with ID starting with 6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4 not found: ID does not exist" containerID="6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4"
Jul 28 20:47:54.316292 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.316254    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4"} err="failed to get container status \"6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4\": rpc error: code = NotFound desc = could not find container \"6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4\": container with ID starting with 6441a4e91861dfaebbc13e3278e1e8de542d8608a518b89344aba2dbc4bc61e4 not found: ID does not exist"
Jul 28 20:47:54.316292 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.316268    2578 scope.go:117] "RemoveContainer" containerID="d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f"
Jul 28 20:47:54.316504 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:47:54.316484    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f\": container with ID starting with d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f not found: ID does not exist" containerID="d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f"
Jul 28 20:47:54.316564 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.316508    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f"} err="failed to get container status \"d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f\": rpc error: code = NotFound desc = could not find container \"d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f\": container with ID starting with d2cfecd7343a8e45da1e34c78e92c76115b9dba860095f92b6a9918ddf2bd84f not found: ID does not exist"
Jul 28 20:47:54.328033 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328011    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 20:47:54.328427 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328414    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy-thanos"
Jul 28 20:47:54.328469 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328429    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy-thanos"
Jul 28 20:47:54.328469 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328440    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="prometheus"
Jul 28 20:47:54.328469 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328445    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="prometheus"
Jul 28 20:47:54.328469 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328452    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="alertmanager"
Jul 28 20:47:54.328469 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328457    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="alertmanager"
Jul 28 20:47:54.328469 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328462    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy-metric"
Jul 28 20:47:54.328469 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328467    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy-metric"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328474    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328479    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328484    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="prom-label-proxy"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328489    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="prom-label-proxy"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328495    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="config-reloader"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328499    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="config-reloader"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328504    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="thanos-sidecar"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328509    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="thanos-sidecar"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328516    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy-web"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328521    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy-web"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328528    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="init-config-reloader"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328533    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="init-config-reloader"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328538    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="config-reloader"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328542    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="config-reloader"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328548    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="init-config-reloader"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328553    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="init-config-reloader"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328558    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328564    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328570    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy-web"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328575    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy-web"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328612    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy-web"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328618    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328624    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328630    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="prometheus"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328635    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="config-reloader"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328640    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="alertmanager"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328645    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy-thanos"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328650    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="kube-rbac-proxy-metric"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328655    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="thanos-sidecar"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328659    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" containerName="kube-rbac-proxy-web"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328665    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="config-reloader"
Jul 28 20:47:54.328659 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.328670    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10340074-7474-46f2-be16-d4defe28a2d3" containerName="prom-label-proxy"
Jul 28 20:47:54.333672 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.333656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.337311 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.337293    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 20:47:54.337421 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.337361    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d9nc223vlqt2u\""
Jul 28 20:47:54.337491 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.337429    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 20:47:54.337549 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.337515    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 20:47:54.337713 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.337699    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 20:47:54.337940 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.337927    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 20:47:54.337998 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.337987    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 20:47:54.338052 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.338002    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 20:47:54.338623 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.338607    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 20:47:54.338869 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.338854    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 20:47:54.338966 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.338868    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 20:47:54.339673 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.339654    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 20:47:54.339760 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.339749    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 20:47:54.341717 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.341662    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 20:47:54.342908 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.342891    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t2gr7\""
Jul 28 20:47:54.344438 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.344422    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 20:47:54.360218 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a32df3a-e587-48f2-95d5-e35eab0b6272-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360303 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360303 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360303 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360438 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360438 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360438 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360438 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360591 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-config\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360591 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360591 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360697 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360697 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360697 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360849 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360849 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360849 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.360849 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.361013 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfp6h\" (UniqueName: \"kubernetes.io/projected/3a32df3a-e587-48f2-95d5-e35eab0b6272-kube-api-access-bfp6h\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.361013 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.360922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a32df3a-e587-48f2-95d5-e35eab0b6272-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.361515 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.361498    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 20:47:54.461758 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.461758 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfp6h\" (UniqueName: \"kubernetes.io/projected/3a32df3a-e587-48f2-95d5-e35eab0b6272-kube-api-access-bfp6h\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.461758 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a32df3a-e587-48f2-95d5-e35eab0b6272-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.461758 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a32df3a-e587-48f2-95d5-e35eab0b6272-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-config\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.461997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462038 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.462036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462586 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.462059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462586 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.462114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462586 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.462144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462586 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.462174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462586 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.462219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462586 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.462525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462586 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.462565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.462946 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.462599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.463440 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.463312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.463440 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.463380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.464672 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.464642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.464808 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.464786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a32df3a-e587-48f2-95d5-e35eab0b6272-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.465678 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.465652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.466025 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.466000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a32df3a-e587-48f2-95d5-e35eab0b6272-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.467141 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.466874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.467141 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.466881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.467141 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.467099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.467340 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.467162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.467340 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.467196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-config\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.467340 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.467244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.467495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.467341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.467495 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.467344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.467603 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.467499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a32df3a-e587-48f2-95d5-e35eab0b6272-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.468245 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.468229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a32df3a-e587-48f2-95d5-e35eab0b6272-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.470889 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.470867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfp6h\" (UniqueName: \"kubernetes.io/projected/3a32df3a-e587-48f2-95d5-e35eab0b6272-kube-api-access-bfp6h\") pod \"prometheus-k8s-0\" (UID: \"3a32df3a-e587-48f2-95d5-e35eab0b6272\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.643577 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.643547    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:47:54.789011 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:54.788982    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 20:47:54.791941 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:47:54.791910    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a32df3a_e587_48f2_95d5_e35eab0b6272.slice/crio-b05fe2bc07884a96efe008d13b59113a48f073d3d6757afdd1b0a7fdd5add9c6 WatchSource:0}: Error finding container b05fe2bc07884a96efe008d13b59113a48f073d3d6757afdd1b0a7fdd5add9c6: Status 404 returned error can't find the container with id b05fe2bc07884a96efe008d13b59113a48f073d3d6757afdd1b0a7fdd5add9c6
Jul 28 20:47:55.269288 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:55.269249    2578 generic.go:356] "Generic (PLEG): container finished" podID="3a32df3a-e587-48f2-95d5-e35eab0b6272" containerID="7a10ff3eef38107ed338acef77e59c74adbc2865cbadc426aff2c5f87f7cdb74" exitCode=0
Jul 28 20:47:55.269707 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:55.269334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a32df3a-e587-48f2-95d5-e35eab0b6272","Type":"ContainerDied","Data":"7a10ff3eef38107ed338acef77e59c74adbc2865cbadc426aff2c5f87f7cdb74"}
Jul 28 20:47:55.269707 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:55.269416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a32df3a-e587-48f2-95d5-e35eab0b6272","Type":"ContainerStarted","Data":"b05fe2bc07884a96efe008d13b59113a48f073d3d6757afdd1b0a7fdd5add9c6"}
Jul 28 20:47:55.530061 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:55.529977    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb6532b-c6e6-4e05-8f07-272b3daa28e5" path="/var/lib/kubelet/pods/bfb6532b-c6e6-4e05-8f07-272b3daa28e5/volumes"
Jul 28 20:47:56.277715 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:56.277684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a32df3a-e587-48f2-95d5-e35eab0b6272","Type":"ContainerStarted","Data":"da18bebd432effd4269d7d71dbaca1acbfdd47200e5a19e41927cf48ed988ae3"}
Jul 28 20:47:56.278114 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:56.277748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a32df3a-e587-48f2-95d5-e35eab0b6272","Type":"ContainerStarted","Data":"b9f10f888246b6a14879f3e1894e2f08676480ff5165e354ca8e895ff1ed0d83"}
Jul 28 20:47:56.278114 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:56.277775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a32df3a-e587-48f2-95d5-e35eab0b6272","Type":"ContainerStarted","Data":"5b2565e5c54662dfe742f0289ad9eaa158c73c0ffab10a7174f454d3fbe2257e"}
Jul 28 20:47:56.278114 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:56.277798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a32df3a-e587-48f2-95d5-e35eab0b6272","Type":"ContainerStarted","Data":"eb0950006ad13def71d9749781011c6a2950c9c3e65391eebea74a5522798ead"}
Jul 28 20:47:56.278114 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:56.277823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a32df3a-e587-48f2-95d5-e35eab0b6272","Type":"ContainerStarted","Data":"3273639dac98636c804c967f62b5a9b0b8deb0512b4e910461c6563da6d5c466"}
Jul 28 20:47:56.278114 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:56.277845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a32df3a-e587-48f2-95d5-e35eab0b6272","Type":"ContainerStarted","Data":"ff1f87d36d4254cc472bcdd4b7bfc7b79b4fb89435618d7886c3fe9e398b3828"}
Jul 28 20:47:56.314007 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:56.313957    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.293295565 podStartE2EDuration="2.313942138s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.646573ms" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:56.312042744 +0000 UTC m=+165.166886724" watchObservedRunningTime="2026-07-28 20:47:56.313942138 +0000 UTC m=+165.168786048"
Jul 28 20:47:59.643814 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:47:59.643779    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:48:54.644737 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:48:54.644704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:48:54.660350 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:48:54.660321    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:48:55.505198 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:48:55.505174    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 20:50:11.441698 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:50:11.441671    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 20:51:08.232358 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.232157    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"]
Jul 28 20:51:08.239346 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.239322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"
Jul 28 20:51:08.242178 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.242155    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:51:08.242479 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.242156    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 20:51:08.242740 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.242219    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-l9j44\""
Jul 28 20:51:08.245465 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.245442    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"]
Jul 28 20:51:08.254837 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.254814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b17fe68-105a-420a-861c-c21406e07420-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qhspc\" (UID: \"7b17fe68-105a-420a-861c-c21406e07420\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"
Jul 28 20:51:08.254939 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.254855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk4hv\" (UniqueName: \"kubernetes.io/projected/7b17fe68-105a-420a-861c-c21406e07420-kube-api-access-vk4hv\") pod \"openshift-lws-operator-fd8ccff4c-qhspc\" (UID: \"7b17fe68-105a-420a-861c-c21406e07420\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"
Jul 28 20:51:08.356284 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.356240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk4hv\" (UniqueName: \"kubernetes.io/projected/7b17fe68-105a-420a-861c-c21406e07420-kube-api-access-vk4hv\") pod \"openshift-lws-operator-fd8ccff4c-qhspc\" (UID: \"7b17fe68-105a-420a-861c-c21406e07420\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"
Jul 28 20:51:08.356466 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.356329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b17fe68-105a-420a-861c-c21406e07420-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qhspc\" (UID: \"7b17fe68-105a-420a-861c-c21406e07420\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"
Jul 28 20:51:08.356660 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.356644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b17fe68-105a-420a-861c-c21406e07420-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qhspc\" (UID: \"7b17fe68-105a-420a-861c-c21406e07420\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"
Jul 28 20:51:08.369936 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.369904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk4hv\" (UniqueName: \"kubernetes.io/projected/7b17fe68-105a-420a-861c-c21406e07420-kube-api-access-vk4hv\") pod \"openshift-lws-operator-fd8ccff4c-qhspc\" (UID: \"7b17fe68-105a-420a-861c-c21406e07420\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"
Jul 28 20:51:08.564423 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.564345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"
Jul 28 20:51:08.690510 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.690482    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc"]
Jul 28 20:51:08.693990 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:51:08.693962    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b17fe68_105a_420a_861c_c21406e07420.slice/crio-41585952dc012fcb2d423a254cfe7d13c34bc0aa474651187024a23419e3ed09 WatchSource:0}: Error finding container 41585952dc012fcb2d423a254cfe7d13c34bc0aa474651187024a23419e3ed09: Status 404 returned error can't find the container with id 41585952dc012fcb2d423a254cfe7d13c34bc0aa474651187024a23419e3ed09
Jul 28 20:51:08.695487 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.695470    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:51:08.966069 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:08.965988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc" event={"ID":"7b17fe68-105a-420a-861c-c21406e07420","Type":"ContainerStarted","Data":"41585952dc012fcb2d423a254cfe7d13c34bc0aa474651187024a23419e3ed09"}
Jul 28 20:51:11.980582 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:11.980549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc" event={"ID":"7b17fe68-105a-420a-861c-c21406e07420","Type":"ContainerStarted","Data":"935c6be00791fe0a7e50459e8e4f4a189c50064f300ad8024d0bcacfa82b383b"}
Jul 28 20:51:11.997565 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:11.997520    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qhspc" podStartSLOduration=1.3962157020000001 podStartE2EDuration="3.997504622s" totalImagesPullingTime="2.60128892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:51:11.995889605 +0000 UTC m=+360.850733516" watchObservedRunningTime="2026-07-28 20:51:11.997504622 +0000 UTC m=+360.852348549"
Jul 28 20:51:31.263803 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.263763    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"]
Jul 28 20:51:31.267217 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.267197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.271066 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.271045    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 20:51:31.271212 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.271065    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 20:51:31.271212 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.271047    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tkb25\""
Jul 28 20:51:31.271212 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.271159    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 20:51:31.288046 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.288016    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"]
Jul 28 20:51:31.343388 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.343362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f0ed41-6296-4bda-ac5c-d421d3c2febe-metrics-cert\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.343565 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.343398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71f0ed41-6296-4bda-ac5c-d421d3c2febe-manager-config\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.343565 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.343449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71f0ed41-6296-4bda-ac5c-d421d3c2febe-cert\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.343565 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.343529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v299c\" (UniqueName: \"kubernetes.io/projected/71f0ed41-6296-4bda-ac5c-d421d3c2febe-kube-api-access-v299c\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.444000 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.443967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v299c\" (UniqueName: \"kubernetes.io/projected/71f0ed41-6296-4bda-ac5c-d421d3c2febe-kube-api-access-v299c\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.444000 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.444002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f0ed41-6296-4bda-ac5c-d421d3c2febe-metrics-cert\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.444287 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.444024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71f0ed41-6296-4bda-ac5c-d421d3c2febe-manager-config\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.444287 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.444096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71f0ed41-6296-4bda-ac5c-d421d3c2febe-cert\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.444665 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.444640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71f0ed41-6296-4bda-ac5c-d421d3c2febe-manager-config\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.446713 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.446686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71f0ed41-6296-4bda-ac5c-d421d3c2febe-cert\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.446824 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.446725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f0ed41-6296-4bda-ac5c-d421d3c2febe-metrics-cert\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.458506 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.458481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v299c\" (UniqueName: \"kubernetes.io/projected/71f0ed41-6296-4bda-ac5c-d421d3c2febe-kube-api-access-v299c\") pod \"lws-controller-manager-57d8fcb7f5-vhplp\" (UID: \"71f0ed41-6296-4bda-ac5c-d421d3c2febe\") " pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.576638 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.576541    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:31.734511 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:31.734426    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"]
Jul 28 20:51:31.736547 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:51:31.736523    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f0ed41_6296_4bda_ac5c_d421d3c2febe.slice/crio-fcc3cff3ea9afee1508f0bafca91155c457df3f9904057c0366e73db159b1bad WatchSource:0}: Error finding container fcc3cff3ea9afee1508f0bafca91155c457df3f9904057c0366e73db159b1bad: Status 404 returned error can't find the container with id fcc3cff3ea9afee1508f0bafca91155c457df3f9904057c0366e73db159b1bad
Jul 28 20:51:32.053782 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.053744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp" event={"ID":"71f0ed41-6296-4bda-ac5c-d421d3c2febe","Type":"ContainerStarted","Data":"fcc3cff3ea9afee1508f0bafca91155c457df3f9904057c0366e73db159b1bad"}
Jul 28 20:51:32.077655 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.077627    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"]
Jul 28 20:51:32.081826 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.081811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.084158 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.084138    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 20:51:32.085209 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.085190    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 20:51:32.085296 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.085209    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 20:51:32.085296 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.085214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gm89g\""
Jul 28 20:51:32.085828 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.085811    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 20:51:32.100775 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.100752    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"]
Jul 28 20:51:32.150845 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.150813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.150980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.150859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxcqd\" (UniqueName: \"kubernetes.io/projected/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-kube-api-access-xxcqd\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.150980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.150879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-tmp\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.150980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.150954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.151129 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.150981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-manifests\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.251794 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.251764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.251950 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.251807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxcqd\" (UniqueName: \"kubernetes.io/projected/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-kube-api-access-xxcqd\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.251950 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.251834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-tmp\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.251950 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.251902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.252159 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.252118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-manifests\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.252538 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.252509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-manifests\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.254349 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.254327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-tmp\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.254578 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.254559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.254693 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.254671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.260759 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.260741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxcqd\" (UniqueName: \"kubernetes.io/projected/0d6e1a1d-9928-4752-97ca-c3aad3802f6f-kube-api-access-xxcqd\") pod \"opendatahub-operator-controller-manager-5966989555-7xnnb\" (UID: \"0d6e1a1d-9928-4752-97ca-c3aad3802f6f\") " pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.391667 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.391582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:32.533951 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:32.533929    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"]
Jul 28 20:51:32.536492 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:51:32.536458    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6e1a1d_9928_4752_97ca_c3aad3802f6f.slice/crio-1ab5633ba63b9053266e1881ea45718aa581d863b2e58d9e5dfc9aba18a1c83c WatchSource:0}: Error finding container 1ab5633ba63b9053266e1881ea45718aa581d863b2e58d9e5dfc9aba18a1c83c: Status 404 returned error can't find the container with id 1ab5633ba63b9053266e1881ea45718aa581d863b2e58d9e5dfc9aba18a1c83c
Jul 28 20:51:33.059909 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:33.059865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb" event={"ID":"0d6e1a1d-9928-4752-97ca-c3aad3802f6f","Type":"ContainerStarted","Data":"1ab5633ba63b9053266e1881ea45718aa581d863b2e58d9e5dfc9aba18a1c83c"}
Jul 28 20:51:34.067024 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:34.066975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp" event={"ID":"71f0ed41-6296-4bda-ac5c-d421d3c2febe","Type":"ContainerStarted","Data":"bf3f844f237fe3ed7cf5cd99f618c1b8b6d4d145cd5c66f009d28a3087e8c9f5"}
Jul 28 20:51:34.067512 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:34.067333    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:34.120502 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:34.120449    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp" podStartSLOduration=1.384983814 podStartE2EDuration="3.120430206s" totalImagesPullingTime="1.735446392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:51:34.11888004 +0000 UTC m=+382.973723952" watchObservedRunningTime="2026-07-28 20:51:34.120430206 +0000 UTC m=+382.975274118"
Jul 28 20:51:36.078279 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:36.078244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb" event={"ID":"0d6e1a1d-9928-4752-97ca-c3aad3802f6f","Type":"ContainerStarted","Data":"f512b4d234ab4bdf886f8727d1e3978315e55443f16f896ebd9e00d463d240bd"}
Jul 28 20:51:37.084189 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:37.084149    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d6e1a1d-9928-4752-97ca-c3aad3802f6f" containerID="f512b4d234ab4bdf886f8727d1e3978315e55443f16f896ebd9e00d463d240bd" exitCode=0
Jul 28 20:51:37.084570 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:37.084239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb" event={"ID":"0d6e1a1d-9928-4752-97ca-c3aad3802f6f","Type":"ContainerDied","Data":"f512b4d234ab4bdf886f8727d1e3978315e55443f16f896ebd9e00d463d240bd"}
Jul 28 20:51:38.090551 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:38.090518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb" event={"ID":"0d6e1a1d-9928-4752-97ca-c3aad3802f6f","Type":"ContainerStarted","Data":"a4a693c14e06c674ee477db933ce699fb8ca6eed9062784c62ab8d85400ef080"}
Jul 28 20:51:38.090973 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:38.090644    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:51:38.132460 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:38.132408    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb" podStartSLOduration=1.647270916 podStartE2EDuration="6.132393069s" totalImagesPullingTime="3.403014581s" totalInitContainerRuntime="1.082107572s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:51:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:51:38.131872529 +0000 UTC m=+386.986716440" watchObservedRunningTime="2026-07-28 20:51:38.132393069 +0000 UTC m=+386.987236980"
Jul 28 20:51:45.074605 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:45.074568    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57d8fcb7f5-vhplp"
Jul 28 20:51:49.096918 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:51:49.096883    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5966989555-7xnnb"
Jul 28 20:52:11.709559 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.709528    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"]
Jul 28 20:52:11.719489 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.719469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.722558 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.722534    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 20:52:11.722691 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.722590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-86zrc\""
Jul 28 20:52:11.722953 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.722934    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 20:52:11.723569 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.723549    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 20:52:11.728017 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.727995    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"]
Jul 28 20:52:11.783758 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.783727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.783758 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.783759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.783975 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.783781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.783975 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.783800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcp79\" (UniqueName: \"kubernetes.io/projected/0684770d-58b4-496c-8a0e-aed1a06baaf8-kube-api-access-hcp79\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.783975 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.783921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.783975 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.783957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0684770d-58b4-496c-8a0e-aed1a06baaf8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.784191 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.783984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.784191 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.784029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.784191 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.784065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885328 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0684770d-58b4-496c-8a0e-aed1a06baaf8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885484 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885484 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885484 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885484 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885484 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885484 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885759 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcp79\" (UniqueName: \"kubernetes.io/projected/0684770d-58b4-496c-8a0e-aed1a06baaf8-kube-api-access-hcp79\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885759 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885759 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885909 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.885909 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885851    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.886017 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.885999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.886167 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.886149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0684770d-58b4-496c-8a0e-aed1a06baaf8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.887992 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.887953    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.888178 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.888160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.893666 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.893639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0684770d-58b4-496c-8a0e-aed1a06baaf8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:11.894043 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:11.894011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcp79\" (UniqueName: \"kubernetes.io/projected/0684770d-58b4-496c-8a0e-aed1a06baaf8-kube-api-access-hcp79\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p\" (UID: \"0684770d-58b4-496c-8a0e-aed1a06baaf8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:12.031535 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:12.031503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:12.167472 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:12.167446    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"]
Jul 28 20:52:12.170058 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:52:12.170015    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0684770d_58b4_496c_8a0e_aed1a06baaf8.slice/crio-41aa13daa9a21ce266dae2aa5d8c4be4a4ee51d0817444ae9c7676c76d6f7dfd WatchSource:0}: Error finding container 41aa13daa9a21ce266dae2aa5d8c4be4a4ee51d0817444ae9c7676c76d6f7dfd: Status 404 returned error can't find the container with id 41aa13daa9a21ce266dae2aa5d8c4be4a4ee51d0817444ae9c7676c76d6f7dfd
Jul 28 20:52:12.224606 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:12.224572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p" event={"ID":"0684770d-58b4-496c-8a0e-aed1a06baaf8","Type":"ContainerStarted","Data":"41aa13daa9a21ce266dae2aa5d8c4be4a4ee51d0817444ae9c7676c76d6f7dfd"}
Jul 28 20:52:15.549066 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:15.549021    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 20:52:15.549329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:15.549130    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 20:52:15.549329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:15.549161    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 20:52:16.242883 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:16.242846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p" event={"ID":"0684770d-58b4-496c-8a0e-aed1a06baaf8","Type":"ContainerStarted","Data":"3bd5d7245416ba310cb851a819fee4115a297a5a27923802e80cca2be5d7d655"}
Jul 28 20:52:16.267656 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:16.267596    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p" podStartSLOduration=1.8914206789999999 podStartE2EDuration="5.267576451s" totalImagesPullingTime="3.376155772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:16.264769301 +0000 UTC m=+425.119613218" watchObservedRunningTime="2026-07-28 20:52:16.267576451 +0000 UTC m=+425.122420363"
Jul 28 20:52:17.031755 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:17.031711    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:17.036240 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:17.036217    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:17.247039 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:17.247008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:17.247916 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:17.247898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p"
Jul 28 20:52:53.030621 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.030544    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-64bct"]
Jul 28 20:52:53.041176 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.041147    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-64bct"
Jul 28 20:52:53.041693 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.041666    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-64bct"]
Jul 28 20:52:53.044207 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.044179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-kmt5p\""
Jul 28 20:52:53.045488 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.045256    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 20:52:53.045488 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.045303    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 20:52:53.126852 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.126823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqz57\" (UniqueName: \"kubernetes.io/projected/38db84fb-0311-4232-abd0-b350d6bb5da2-kube-api-access-vqz57\") pod \"kuadrant-operator-catalog-64bct\" (UID: \"38db84fb-0311-4232-abd0-b350d6bb5da2\") " pod="kuadrant-system/kuadrant-operator-catalog-64bct"
Jul 28 20:52:53.228145 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.228109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqz57\" (UniqueName: \"kubernetes.io/projected/38db84fb-0311-4232-abd0-b350d6bb5da2-kube-api-access-vqz57\") pod \"kuadrant-operator-catalog-64bct\" (UID: \"38db84fb-0311-4232-abd0-b350d6bb5da2\") " pod="kuadrant-system/kuadrant-operator-catalog-64bct"
Jul 28 20:52:53.236746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.236721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqz57\" (UniqueName: \"kubernetes.io/projected/38db84fb-0311-4232-abd0-b350d6bb5da2-kube-api-access-vqz57\") pod \"kuadrant-operator-catalog-64bct\" (UID: \"38db84fb-0311-4232-abd0-b350d6bb5da2\") " pod="kuadrant-system/kuadrant-operator-catalog-64bct"
Jul 28 20:52:53.351674 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.351598    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-64bct"
Jul 28 20:52:53.394062 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.394028    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-64bct"]
Jul 28 20:52:53.477573 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.477489    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-64bct"]
Jul 28 20:52:53.480128 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:52:53.480096    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38db84fb_0311_4232_abd0_b350d6bb5da2.slice/crio-79509c78778dbb19666ab402191484d86a59b2789d5fe9b72a37aae7a9cecd3a WatchSource:0}: Error finding container 79509c78778dbb19666ab402191484d86a59b2789d5fe9b72a37aae7a9cecd3a: Status 404 returned error can't find the container with id 79509c78778dbb19666ab402191484d86a59b2789d5fe9b72a37aae7a9cecd3a
Jul 28 20:52:53.605168 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.605100    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8vvr6"]
Jul 28 20:52:53.609886 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.609869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8vvr6"
Jul 28 20:52:53.614786 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.614766    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8vvr6"]
Jul 28 20:52:53.732557 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.732523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcndl\" (UniqueName: \"kubernetes.io/projected/ca15543a-555e-4a8b-9e8b-d7d9c4e30899-kube-api-access-fcndl\") pod \"kuadrant-operator-catalog-8vvr6\" (UID: \"ca15543a-555e-4a8b-9e8b-d7d9c4e30899\") " pod="kuadrant-system/kuadrant-operator-catalog-8vvr6"
Jul 28 20:52:53.833585 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.833558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcndl\" (UniqueName: \"kubernetes.io/projected/ca15543a-555e-4a8b-9e8b-d7d9c4e30899-kube-api-access-fcndl\") pod \"kuadrant-operator-catalog-8vvr6\" (UID: \"ca15543a-555e-4a8b-9e8b-d7d9c4e30899\") " pod="kuadrant-system/kuadrant-operator-catalog-8vvr6"
Jul 28 20:52:53.841944 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.841914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcndl\" (UniqueName: \"kubernetes.io/projected/ca15543a-555e-4a8b-9e8b-d7d9c4e30899-kube-api-access-fcndl\") pod \"kuadrant-operator-catalog-8vvr6\" (UID: \"ca15543a-555e-4a8b-9e8b-d7d9c4e30899\") " pod="kuadrant-system/kuadrant-operator-catalog-8vvr6"
Jul 28 20:52:53.919476 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:53.919414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8vvr6"
Jul 28 20:52:54.052743 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:54.052706    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8vvr6"]
Jul 28 20:52:54.086055 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:52:54.086021    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca15543a_555e_4a8b_9e8b_d7d9c4e30899.slice/crio-6662809d532d6842a38398e83c2639ee3d99d8e15df1afb273784f54fe083a41 WatchSource:0}: Error finding container 6662809d532d6842a38398e83c2639ee3d99d8e15df1afb273784f54fe083a41: Status 404 returned error can't find the container with id 6662809d532d6842a38398e83c2639ee3d99d8e15df1afb273784f54fe083a41
Jul 28 20:52:54.395338 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:54.395302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8vvr6" event={"ID":"ca15543a-555e-4a8b-9e8b-d7d9c4e30899","Type":"ContainerStarted","Data":"6662809d532d6842a38398e83c2639ee3d99d8e15df1afb273784f54fe083a41"}
Jul 28 20:52:54.396638 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:54.396611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-64bct" event={"ID":"38db84fb-0311-4232-abd0-b350d6bb5da2","Type":"ContainerStarted","Data":"79509c78778dbb19666ab402191484d86a59b2789d5fe9b72a37aae7a9cecd3a"}
Jul 28 20:52:56.408124 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:56.408071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-64bct" event={"ID":"38db84fb-0311-4232-abd0-b350d6bb5da2","Type":"ContainerStarted","Data":"2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332"}
Jul 28 20:52:56.408605 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:56.408118    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-64bct" podUID="38db84fb-0311-4232-abd0-b350d6bb5da2" containerName="registry-server" containerID="cri-o://2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332" gracePeriod=2
Jul 28 20:52:56.409715 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:56.409689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8vvr6" event={"ID":"ca15543a-555e-4a8b-9e8b-d7d9c4e30899","Type":"ContainerStarted","Data":"d1553fa6f6acfabf8ee54c316283248feeabad8f6f70fb55242a5c1de3ad1d00"}
Jul 28 20:52:56.426090 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:56.426012    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-64bct" podStartSLOduration=1.400382735 podStartE2EDuration="3.425996146s" totalImagesPullingTime="2.025613411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:56.42511242 +0000 UTC m=+465.279956352" watchObservedRunningTime="2026-07-28 20:52:56.425996146 +0000 UTC m=+465.280840118"
Jul 28 20:52:56.446401 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:56.446345    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8vvr6" podStartSLOduration=1.963409996 podStartE2EDuration="3.44633225s" totalImagesPullingTime="1.482922254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:56.445730375 +0000 UTC m=+465.300574297" watchObservedRunningTime="2026-07-28 20:52:56.44633225 +0000 UTC m=+465.301176160"
Jul 28 20:52:56.653033 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:56.653011    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-64bct"
Jul 28 20:52:56.758583 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:56.758554    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqz57\" (UniqueName: \"kubernetes.io/projected/38db84fb-0311-4232-abd0-b350d6bb5da2-kube-api-access-vqz57\") pod \"38db84fb-0311-4232-abd0-b350d6bb5da2\" (UID: \"38db84fb-0311-4232-abd0-b350d6bb5da2\") "
Jul 28 20:52:56.760812 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:56.760779    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38db84fb-0311-4232-abd0-b350d6bb5da2-kube-api-access-vqz57" (OuterVolumeSpecName: "kube-api-access-vqz57") pod "38db84fb-0311-4232-abd0-b350d6bb5da2" (UID: "38db84fb-0311-4232-abd0-b350d6bb5da2"). InnerVolumeSpecName "kube-api-access-vqz57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:52:56.859380 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:56.859347    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqz57\" (UniqueName: \"kubernetes.io/projected/38db84fb-0311-4232-abd0-b350d6bb5da2-kube-api-access-vqz57\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:52:57.416240 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:57.416208    2578 generic.go:356] "Generic (PLEG): container finished" podID="38db84fb-0311-4232-abd0-b350d6bb5da2" containerID="2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332" exitCode=0
Jul 28 20:52:57.416686 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:57.416251    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-64bct"
Jul 28 20:52:57.416686 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:57.416291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-64bct" event={"ID":"38db84fb-0311-4232-abd0-b350d6bb5da2","Type":"ContainerDied","Data":"2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332"}
Jul 28 20:52:57.416686 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:57.416340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-64bct" event={"ID":"38db84fb-0311-4232-abd0-b350d6bb5da2","Type":"ContainerDied","Data":"79509c78778dbb19666ab402191484d86a59b2789d5fe9b72a37aae7a9cecd3a"}
Jul 28 20:52:57.416686 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:57.416360    2578 scope.go:117] "RemoveContainer" containerID="2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332"
Jul 28 20:52:57.425553 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:57.425532    2578 scope.go:117] "RemoveContainer" containerID="2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332"
Jul 28 20:52:57.425809 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:52:57.425794    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332\": container with ID starting with 2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332 not found: ID does not exist" containerID="2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332"
Jul 28 20:52:57.425877 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:57.425817    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332"} err="failed to get container status \"2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332\": rpc error: code = NotFound desc = could not find container \"2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332\": container with ID starting with 2a09d3cb83a32ea4be265e1cf76d85eb9a3d030d537811c0a4e94b9a0c90d332 not found: ID does not exist"
Jul 28 20:52:57.438238 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:57.438215    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-64bct"]
Jul 28 20:52:57.442573 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:57.442556    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-64bct"]
Jul 28 20:52:57.528017 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:52:57.527990    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38db84fb-0311-4232-abd0-b350d6bb5da2" path="/var/lib/kubelet/pods/38db84fb-0311-4232-abd0-b350d6bb5da2/volumes"
Jul 28 20:53:03.919967 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:03.919929    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-8vvr6"
Jul 28 20:53:03.920501 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:03.919997    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-8vvr6"
Jul 28 20:53:03.941755 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:03.941729    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-8vvr6"
Jul 28 20:53:04.466466 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:04.466438    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-8vvr6"
Jul 28 20:53:24.896259 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:24.896226    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf"]
Jul 28 20:53:24.896670 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:24.896544    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38db84fb-0311-4232-abd0-b350d6bb5da2" containerName="registry-server"
Jul 28 20:53:24.896670 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:24.896554    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38db84fb-0311-4232-abd0-b350d6bb5da2" containerName="registry-server"
Jul 28 20:53:24.896670 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:24.896608    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="38db84fb-0311-4232-abd0-b350d6bb5da2" containerName="registry-server"
Jul 28 20:53:24.903194 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:24.903172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf"
Jul 28 20:53:24.905968 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:24.905944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-w77zn\""
Jul 28 20:53:24.906113 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:24.905945    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 28 20:53:24.909512 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:24.909485    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf"]
Jul 28 20:53:24.979044 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:24.979016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbhm8\" (UniqueName: \"kubernetes.io/projected/66d2cdfb-a082-48d8-83db-571d37580d75-kube-api-access-tbhm8\") pod \"dns-operator-controller-manager-864dd78c4c-4dbnf\" (UID: \"66d2cdfb-a082-48d8-83db-571d37580d75\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf"
Jul 28 20:53:25.079754 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:25.079716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbhm8\" (UniqueName: \"kubernetes.io/projected/66d2cdfb-a082-48d8-83db-571d37580d75-kube-api-access-tbhm8\") pod \"dns-operator-controller-manager-864dd78c4c-4dbnf\" (UID: \"66d2cdfb-a082-48d8-83db-571d37580d75\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf"
Jul 28 20:53:25.091330 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:25.091307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbhm8\" (UniqueName: \"kubernetes.io/projected/66d2cdfb-a082-48d8-83db-571d37580d75-kube-api-access-tbhm8\") pod \"dns-operator-controller-manager-864dd78c4c-4dbnf\" (UID: \"66d2cdfb-a082-48d8-83db-571d37580d75\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf"
Jul 28 20:53:25.214177 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:25.214070    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf"
Jul 28 20:53:25.344546 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:25.344522    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf"]
Jul 28 20:53:25.346888 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:53:25.346862    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d2cdfb_a082_48d8_83db_571d37580d75.slice/crio-9c3e122fbfcca59db3be34e980fb0fb352a14ff52dd63a706c99abbb4a30cdb6 WatchSource:0}: Error finding container 9c3e122fbfcca59db3be34e980fb0fb352a14ff52dd63a706c99abbb4a30cdb6: Status 404 returned error can't find the container with id 9c3e122fbfcca59db3be34e980fb0fb352a14ff52dd63a706c99abbb4a30cdb6
Jul 28 20:53:25.536068 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:25.536041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf" event={"ID":"66d2cdfb-a082-48d8-83db-571d37580d75","Type":"ContainerStarted","Data":"9c3e122fbfcca59db3be34e980fb0fb352a14ff52dd63a706c99abbb4a30cdb6"}
Jul 28 20:53:28.548369 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:28.548331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf" event={"ID":"66d2cdfb-a082-48d8-83db-571d37580d75","Type":"ContainerStarted","Data":"6cdf726605c62784a920788a1ea530af1853ff3fd563d599c411fdccdc84b95a"}
Jul 28 20:53:28.548759 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:28.548432    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf"
Jul 28 20:53:28.569702 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:28.569620    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf" podStartSLOduration=2.423762795 podStartE2EDuration="4.569600714s" totalImagesPullingTime="2.145837919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:28.56933156 +0000 UTC m=+497.424175472" watchObservedRunningTime="2026-07-28 20:53:28.569600714 +0000 UTC m=+497.424444625"
Jul 28 20:53:30.262100 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.252611    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"]
Jul 28 20:53:30.268950 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.268918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:30.270002 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.269808    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"]
Jul 28 20:53:30.272328 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.272309    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-znxx8\""
Jul 28 20:53:30.322711 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.322686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" (UID: \"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:30.322854 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.322717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpqf\" (UniqueName: \"kubernetes.io/projected/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-kube-api-access-nxpqf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" (UID: \"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:30.423494 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.423458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" (UID: \"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:30.424048 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.423873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpqf\" (UniqueName: \"kubernetes.io/projected/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-kube-api-access-nxpqf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" (UID: \"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:30.428113 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.424389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" (UID: \"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:30.433143 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.433115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpqf\" (UniqueName: \"kubernetes.io/projected/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-kube-api-access-nxpqf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" (UID: \"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:30.579863 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.579781    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:30.719969 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:30.719945    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"]
Jul 28 20:53:30.722669 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:53:30.722642    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a4ffc2_60e7_4f27_adeb_d3dc81b7d01d.slice/crio-def58af7d8652d9a6bea691a8c9cc66624411e45624e9dfdb48b32c88a772002 WatchSource:0}: Error finding container def58af7d8652d9a6bea691a8c9cc66624411e45624e9dfdb48b32c88a772002: Status 404 returned error can't find the container with id def58af7d8652d9a6bea691a8c9cc66624411e45624e9dfdb48b32c88a772002
Jul 28 20:53:31.560506 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:31.560470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd" event={"ID":"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d","Type":"ContainerStarted","Data":"def58af7d8652d9a6bea691a8c9cc66624411e45624e9dfdb48b32c88a772002"}
Jul 28 20:53:33.680184 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:33.680151    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt"]
Jul 28 20:53:33.683930 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:33.683903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt"
Jul 28 20:53:33.686596 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:33.686570    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bz6r2\""
Jul 28 20:53:33.696280 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:33.696259    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt"]
Jul 28 20:53:33.752335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:33.752297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxwgr\" (UniqueName: \"kubernetes.io/projected/8206184c-abd7-4c3f-8235-e27f2215850b-kube-api-access-jxwgr\") pod \"authorino-operator-75cc8d5cb7-fkpjt\" (UID: \"8206184c-abd7-4c3f-8235-e27f2215850b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt"
Jul 28 20:53:33.853533 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:33.853497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxwgr\" (UniqueName: \"kubernetes.io/projected/8206184c-abd7-4c3f-8235-e27f2215850b-kube-api-access-jxwgr\") pod \"authorino-operator-75cc8d5cb7-fkpjt\" (UID: \"8206184c-abd7-4c3f-8235-e27f2215850b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt"
Jul 28 20:53:33.865287 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:33.865261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxwgr\" (UniqueName: \"kubernetes.io/projected/8206184c-abd7-4c3f-8235-e27f2215850b-kube-api-access-jxwgr\") pod \"authorino-operator-75cc8d5cb7-fkpjt\" (UID: \"8206184c-abd7-4c3f-8235-e27f2215850b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt"
Jul 28 20:53:33.996919 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:33.996891    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt"
Jul 28 20:53:34.147689 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:34.147661    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt"]
Jul 28 20:53:34.150165 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:53:34.150140    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8206184c_abd7_4c3f_8235_e27f2215850b.slice/crio-f51079d3b444cdb53010943fd5fcf22dc315eebb50911744ccaae851875afeb4 WatchSource:0}: Error finding container f51079d3b444cdb53010943fd5fcf22dc315eebb50911744ccaae851875afeb4: Status 404 returned error can't find the container with id f51079d3b444cdb53010943fd5fcf22dc315eebb50911744ccaae851875afeb4
Jul 28 20:53:34.575152 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:34.575119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt" event={"ID":"8206184c-abd7-4c3f-8235-e27f2215850b","Type":"ContainerStarted","Data":"f51079d3b444cdb53010943fd5fcf22dc315eebb50911744ccaae851875afeb4"}
Jul 28 20:53:38.595298 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:38.595264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt" event={"ID":"8206184c-abd7-4c3f-8235-e27f2215850b","Type":"ContainerStarted","Data":"11ba98eb6c8ed1d2b26479442e4722a5bc66ef342b88b11b7a3e691bcddd3e48"}
Jul 28 20:53:38.595748 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:38.595326    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt"
Jul 28 20:53:38.596769 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:38.596746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd" event={"ID":"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d","Type":"ContainerStarted","Data":"898c7cd712001f3c750d087e4eec956689cbc5a21c40a2d8e6f950bf1d491675"}
Jul 28 20:53:38.596879 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:38.596869    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:38.614857 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:38.614815    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt" podStartSLOduration=2.050575743 podStartE2EDuration="5.614795356s" totalImagesPullingTime="3.564219613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:38.613670153 +0000 UTC m=+507.468514089" watchObservedRunningTime="2026-07-28 20:53:38.614795356 +0000 UTC m=+507.469639337"
Jul 28 20:53:38.636324 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:38.636283    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd" podStartSLOduration=1.6340326520000001 podStartE2EDuration="8.636270954s" totalImagesPullingTime="7.002238302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:38.634905798 +0000 UTC m=+507.489749710" watchObservedRunningTime="2026-07-28 20:53:38.636270954 +0000 UTC m=+507.491114865"
Jul 28 20:53:39.554153 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:39.554129    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dbnf"
Jul 28 20:53:49.603648 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:49.603606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fkpjt"
Jul 28 20:53:49.604121 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:49.603700    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:51.164709 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.164667    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"]
Jul 28 20:53:51.168303 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.168288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.190901 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.190877    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"]
Jul 28 20:53:51.311137 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.311100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37ffd50c-d238-4643-8542-a3683608729e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" (UID: \"37ffd50c-d238-4643-8542-a3683608729e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.311333 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.311144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vjtf\" (UniqueName: \"kubernetes.io/projected/37ffd50c-d238-4643-8542-a3683608729e-kube-api-access-5vjtf\") pod \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" (UID: \"37ffd50c-d238-4643-8542-a3683608729e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.412680 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.412645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37ffd50c-d238-4643-8542-a3683608729e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" (UID: \"37ffd50c-d238-4643-8542-a3683608729e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.412680 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.412685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vjtf\" (UniqueName: \"kubernetes.io/projected/37ffd50c-d238-4643-8542-a3683608729e-kube-api-access-5vjtf\") pod \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" (UID: \"37ffd50c-d238-4643-8542-a3683608729e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.413030 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.413011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37ffd50c-d238-4643-8542-a3683608729e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" (UID: \"37ffd50c-d238-4643-8542-a3683608729e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.424695 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.424629    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"]
Jul 28 20:53:51.424898 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:51.424879    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5vjtf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn" podUID="37ffd50c-d238-4643-8542-a3683608729e"
Jul 28 20:53:51.430405 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.430384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vjtf\" (UniqueName: \"kubernetes.io/projected/37ffd50c-d238-4643-8542-a3683608729e-kube-api-access-5vjtf\") pod \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" (UID: \"37ffd50c-d238-4643-8542-a3683608729e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.433999 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.433977    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"]
Jul 28 20:53:51.445757 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.445725    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"]
Jul 28 20:53:51.445922 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.445896    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd" podUID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" containerName="manager" containerID="cri-o://898c7cd712001f3c750d087e4eec956689cbc5a21c40a2d8e6f950bf1d491675" gracePeriod=2
Jul 28 20:53:51.450663 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.450637    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"]
Jul 28 20:53:51.454529 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.454507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:53:51.468589 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.468570    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"]
Jul 28 20:53:51.472552 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.472530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"]
Jul 28 20:53:51.480434 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:51.480400    2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\": pods \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:51.482661 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.482638    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"]
Jul 28 20:53:51.482740 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:51.482641    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" is forbidden: User \"system:node:ip-10-0-138-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-164.ec2.internal' and this object" podUID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:51.483033 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.483018    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" containerName="manager"
Jul 28 20:53:51.483101 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.483035    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" containerName="manager"
Jul 28 20:53:51.483140 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.483135    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" containerName="manager"
Jul 28 20:53:51.486008 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.485988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:53:51.503125 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.503094    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"]
Jul 28 20:53:51.547516 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:51.547485    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" is forbidden: User \"system:node:ip-10-0-138-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-164.ec2.internal' and this object" podUID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:51.613545 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.613512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ths\" (UniqueName: \"kubernetes.io/projected/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-kube-api-access-b2ths\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gnlnw\" (UID: \"c45c31c9-7e31-44a6-b9ad-b09a78ee603c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:53:51.613725 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.613591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f2208e9-f94e-4aef-a584-3b0cf75a904c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6svf5\" (UID: \"3f2208e9-f94e-4aef-a584-3b0cf75a904c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:53:51.613725 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.613628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gnlnw\" (UID: \"c45c31c9-7e31-44a6-b9ad-b09a78ee603c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:53:51.613854 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.613731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slr92\" (UniqueName: \"kubernetes.io/projected/3f2208e9-f94e-4aef-a584-3b0cf75a904c-kube-api-access-slr92\") pod \"kuadrant-operator-controller-manager-55467d9f7-6svf5\" (UID: \"3f2208e9-f94e-4aef-a584-3b0cf75a904c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:53:51.656808 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.656779    2578 generic.go:356] "Generic (PLEG): container finished" podID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" containerID="898c7cd712001f3c750d087e4eec956689cbc5a21c40a2d8e6f950bf1d491675" exitCode=0
Jul 28 20:53:51.656960 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.656884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.659631 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:51.659602    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" is forbidden: User \"system:node:ip-10-0-138-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-164.ec2.internal' and this object" podUID="37ffd50c-d238-4643-8542-a3683608729e" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.663445 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.663423    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.666364 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:51.666336    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" is forbidden: User \"system:node:ip-10-0-138-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-164.ec2.internal' and this object" podUID="37ffd50c-d238-4643-8542-a3683608729e" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.687216 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.687169    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:51.689859 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:51.689835    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" is forbidden: User \"system:node:ip-10-0-138-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-164.ec2.internal' and this object" podUID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:51.692221 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:51.692199    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" is forbidden: User \"system:node:ip-10-0-138-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-164.ec2.internal' and this object" podUID="37ffd50c-d238-4643-8542-a3683608729e" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:51.714607 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.714583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slr92\" (UniqueName: \"kubernetes.io/projected/3f2208e9-f94e-4aef-a584-3b0cf75a904c-kube-api-access-slr92\") pod \"kuadrant-operator-controller-manager-55467d9f7-6svf5\" (UID: \"3f2208e9-f94e-4aef-a584-3b0cf75a904c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:53:51.714707 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.714652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ths\" (UniqueName: \"kubernetes.io/projected/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-kube-api-access-b2ths\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gnlnw\" (UID: \"c45c31c9-7e31-44a6-b9ad-b09a78ee603c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:53:51.714707 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.714675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f2208e9-f94e-4aef-a584-3b0cf75a904c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6svf5\" (UID: \"3f2208e9-f94e-4aef-a584-3b0cf75a904c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:53:51.714707 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.714704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gnlnw\" (UID: \"c45c31c9-7e31-44a6-b9ad-b09a78ee603c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:53:51.715045 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.715029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gnlnw\" (UID: \"c45c31c9-7e31-44a6-b9ad-b09a78ee603c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:53:51.715045 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.715037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f2208e9-f94e-4aef-a584-3b0cf75a904c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6svf5\" (UID: \"3f2208e9-f94e-4aef-a584-3b0cf75a904c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:53:51.739623 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.739596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slr92\" (UniqueName: \"kubernetes.io/projected/3f2208e9-f94e-4aef-a584-3b0cf75a904c-kube-api-access-slr92\") pod \"kuadrant-operator-controller-manager-55467d9f7-6svf5\" (UID: \"3f2208e9-f94e-4aef-a584-3b0cf75a904c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:53:51.742587 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.742562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ths\" (UniqueName: \"kubernetes.io/projected/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-kube-api-access-b2ths\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gnlnw\" (UID: \"c45c31c9-7e31-44a6-b9ad-b09a78ee603c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:53:51.815430 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.815401    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vjtf\" (UniqueName: \"kubernetes.io/projected/37ffd50c-d238-4643-8542-a3683608729e-kube-api-access-5vjtf\") pod \"37ffd50c-d238-4643-8542-a3683608729e\" (UID: \"37ffd50c-d238-4643-8542-a3683608729e\") "
Jul 28 20:53:51.815571 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.815550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37ffd50c-d238-4643-8542-a3683608729e-extensions-socket-volume\") pod \"37ffd50c-d238-4643-8542-a3683608729e\" (UID: \"37ffd50c-d238-4643-8542-a3683608729e\") "
Jul 28 20:53:51.815628 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.815581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxpqf\" (UniqueName: \"kubernetes.io/projected/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-kube-api-access-nxpqf\") pod \"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\" (UID: \"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\") "
Jul 28 20:53:51.815628 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.815606    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-extensions-socket-volume\") pod \"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\" (UID: \"41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d\") "
Jul 28 20:53:51.815850 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.815820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ffd50c-d238-4643-8542-a3683608729e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "37ffd50c-d238-4643-8542-a3683608729e" (UID: "37ffd50c-d238-4643-8542-a3683608729e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:51.816199 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.816144    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" (UID: "41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:51.817548 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.817522    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ffd50c-d238-4643-8542-a3683608729e-kube-api-access-5vjtf" (OuterVolumeSpecName: "kube-api-access-5vjtf") pod "37ffd50c-d238-4643-8542-a3683608729e" (UID: "37ffd50c-d238-4643-8542-a3683608729e"). InnerVolumeSpecName "kube-api-access-5vjtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:53:51.817632 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.817604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-kube-api-access-nxpqf" (OuterVolumeSpecName: "kube-api-access-nxpqf") pod "41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" (UID: "41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d"). InnerVolumeSpecName "kube-api-access-nxpqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:53:51.820745 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.820729    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:53:51.827498 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.827478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:53:51.916913 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.916809    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37ffd50c-d238-4643-8542-a3683608729e-extensions-socket-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:53:51.916913 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.916838    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxpqf\" (UniqueName: \"kubernetes.io/projected/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-kube-api-access-nxpqf\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:53:51.916913 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.916854    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d-extensions-socket-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:53:51.916913 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.916864    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vjtf\" (UniqueName: \"kubernetes.io/projected/37ffd50c-d238-4643-8542-a3683608729e-kube-api-access-5vjtf\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:53:51.968628 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.968600    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"]
Jul 28 20:53:51.971572 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:53:51.971546    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2208e9_f94e_4aef_a584_3b0cf75a904c.slice/crio-abd2033000d08191df1a711da9bf1006ab28b77cd1e042825452a5c863272626 WatchSource:0}: Error finding container abd2033000d08191df1a711da9bf1006ab28b77cd1e042825452a5c863272626: Status 404 returned error can't find the container with id abd2033000d08191df1a711da9bf1006ab28b77cd1e042825452a5c863272626
Jul 28 20:53:51.987174 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:51.987152    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"]
Jul 28 20:53:51.989042 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:53:51.989021    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45c31c9_7e31_44a6_b9ad_b09a78ee603c.slice/crio-1d3b650c2a39be9ea3c5bfefebc918457b54835fa8756affa667bf04b871a02a WatchSource:0}: Error finding container 1d3b650c2a39be9ea3c5bfefebc918457b54835fa8756affa667bf04b871a02a: Status 404 returned error can't find the container with id 1d3b650c2a39be9ea3c5bfefebc918457b54835fa8756affa667bf04b871a02a
Jul 28 20:53:52.662187 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.662147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw" event={"ID":"c45c31c9-7e31-44a6-b9ad-b09a78ee603c","Type":"ContainerStarted","Data":"9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71"}
Jul 28 20:53:52.662646 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.662239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw" event={"ID":"c45c31c9-7e31-44a6-b9ad-b09a78ee603c","Type":"ContainerStarted","Data":"1d3b650c2a39be9ea3c5bfefebc918457b54835fa8756affa667bf04b871a02a"}
Jul 28 20:53:52.662646 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.662270    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:53:52.663727 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.663702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5" event={"ID":"3f2208e9-f94e-4aef-a584-3b0cf75a904c","Type":"ContainerStarted","Data":"54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395"}
Jul 28 20:53:52.663812 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.663744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5" event={"ID":"3f2208e9-f94e-4aef-a584-3b0cf75a904c","Type":"ContainerStarted","Data":"abd2033000d08191df1a711da9bf1006ab28b77cd1e042825452a5c863272626"}
Jul 28 20:53:52.663812 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.663768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:53:52.664938 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.664920    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:52.664979 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.664959    2578 scope.go:117] "RemoveContainer" containerID="898c7cd712001f3c750d087e4eec956689cbc5a21c40a2d8e6f950bf1d491675"
Jul 28 20:53:52.665106 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.665092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:52.691422 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.691337    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw" podStartSLOduration=1.6913189389999999 podStartE2EDuration="1.691318939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:52.688566185 +0000 UTC m=+521.543410097" watchObservedRunningTime="2026-07-28 20:53:52.691318939 +0000 UTC m=+521.546162864"
Jul 28 20:53:52.691552 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:52.691503    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" is forbidden: User \"system:node:ip-10-0-138-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-164.ec2.internal' and this object" podUID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:52.694715 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:52.694693    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" is forbidden: User \"system:node:ip-10-0-138-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-164.ec2.internal' and this object" podUID="37ffd50c-d238-4643-8542-a3683608729e" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:52.723879 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:52.723842    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5" podStartSLOduration=1.723828198 podStartE2EDuration="1.723828198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:52.722713445 +0000 UTC m=+521.577557356" watchObservedRunningTime="2026-07-28 20:53:52.723828198 +0000 UTC m=+521.578672184"
Jul 28 20:53:52.725055 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:52.725023    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wxqfd\" is forbidden: User \"system:node:ip-10-0-138-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-164.ec2.internal' and this object" podUID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wxqfd"
Jul 28 20:53:52.727211 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:53:52.727188    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-tq8sn\" is forbidden: User \"system:node:ip-10-0-138-164.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-164.ec2.internal' and this object" podUID="37ffd50c-d238-4643-8542-a3683608729e" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tq8sn"
Jul 28 20:53:53.528938 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:53.528901    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ffd50c-d238-4643-8542-a3683608729e" path="/var/lib/kubelet/pods/37ffd50c-d238-4643-8542-a3683608729e/volumes"
Jul 28 20:53:53.529176 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:53:53.529163    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d" path="/var/lib/kubelet/pods/41a4ffc2-60e7-4f27-adeb-d3dc81b7d01d/volumes"
Jul 28 20:54:03.673464 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:03.673376    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:54:03.673464 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:03.673450    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:54:03.744315 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:03.744275    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"]
Jul 28 20:54:03.744573 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:03.744509    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw" podUID="c45c31c9-7e31-44a6-b9ad-b09a78ee603c" containerName="manager" containerID="cri-o://9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71" gracePeriod=10
Jul 28 20:54:03.991239 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:03.991213    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:54:04.016065 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.016034    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"]
Jul 28 20:54:04.016414 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.016399    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c45c31c9-7e31-44a6-b9ad-b09a78ee603c" containerName="manager"
Jul 28 20:54:04.016414 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.016416    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c45c31c9-7e31-44a6-b9ad-b09a78ee603c" containerName="manager"
Jul 28 20:54:04.016511 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.016467    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c45c31c9-7e31-44a6-b9ad-b09a78ee603c" containerName="manager"
Jul 28 20:54:04.019335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.019313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 20:54:04.020496 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.020465    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2ths\" (UniqueName: \"kubernetes.io/projected/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-kube-api-access-b2ths\") pod \"c45c31c9-7e31-44a6-b9ad-b09a78ee603c\" (UID: \"c45c31c9-7e31-44a6-b9ad-b09a78ee603c\") "
Jul 28 20:54:04.020595 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.020519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-extensions-socket-volume\") pod \"c45c31c9-7e31-44a6-b9ad-b09a78ee603c\" (UID: \"c45c31c9-7e31-44a6-b9ad-b09a78ee603c\") "
Jul 28 20:54:04.020904 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.020874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c45c31c9-7e31-44a6-b9ad-b09a78ee603c" (UID: "c45c31c9-7e31-44a6-b9ad-b09a78ee603c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:54:04.022877 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.022849    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-kube-api-access-b2ths" (OuterVolumeSpecName: "kube-api-access-b2ths") pod "c45c31c9-7e31-44a6-b9ad-b09a78ee603c" (UID: "c45c31c9-7e31-44a6-b9ad-b09a78ee603c"). InnerVolumeSpecName "kube-api-access-b2ths". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:54:04.037300 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.037273    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"]
Jul 28 20:54:04.121190 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.121152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkcv\" (UniqueName: \"kubernetes.io/projected/fc2b74ec-2180-46c5-91d4-c23682a11a07-kube-api-access-dwkcv\") pod \"kuadrant-operator-controller-manager-86878c769f-94tzq\" (UID: \"fc2b74ec-2180-46c5-91d4-c23682a11a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 20:54:04.121373 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.121220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc2b74ec-2180-46c5-91d4-c23682a11a07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-94tzq\" (UID: \"fc2b74ec-2180-46c5-91d4-c23682a11a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 20:54:04.121373 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.121297    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2ths\" (UniqueName: \"kubernetes.io/projected/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-kube-api-access-b2ths\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:54:04.121373 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.121320    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c45c31c9-7e31-44a6-b9ad-b09a78ee603c-extensions-socket-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:54:04.222321 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.222289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkcv\" (UniqueName: \"kubernetes.io/projected/fc2b74ec-2180-46c5-91d4-c23682a11a07-kube-api-access-dwkcv\") pod \"kuadrant-operator-controller-manager-86878c769f-94tzq\" (UID: \"fc2b74ec-2180-46c5-91d4-c23682a11a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 20:54:04.222479 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.222358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc2b74ec-2180-46c5-91d4-c23682a11a07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-94tzq\" (UID: \"fc2b74ec-2180-46c5-91d4-c23682a11a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 20:54:04.222700 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.222685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc2b74ec-2180-46c5-91d4-c23682a11a07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-94tzq\" (UID: \"fc2b74ec-2180-46c5-91d4-c23682a11a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 20:54:04.230825 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.230794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkcv\" (UniqueName: \"kubernetes.io/projected/fc2b74ec-2180-46c5-91d4-c23682a11a07-kube-api-access-dwkcv\") pod \"kuadrant-operator-controller-manager-86878c769f-94tzq\" (UID: \"fc2b74ec-2180-46c5-91d4-c23682a11a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 20:54:04.337418 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.337377    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 20:54:04.477474 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.477449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"]
Jul 28 20:54:04.479309 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:54:04.479286    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2b74ec_2180_46c5_91d4_c23682a11a07.slice/crio-a81d93749dbe6b883ea2ae85b3b49f02c59b328895ae3860a49f392f84d7073f WatchSource:0}: Error finding container a81d93749dbe6b883ea2ae85b3b49f02c59b328895ae3860a49f392f84d7073f: Status 404 returned error can't find the container with id a81d93749dbe6b883ea2ae85b3b49f02c59b328895ae3860a49f392f84d7073f
Jul 28 20:54:04.723015 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.722977    2578 generic.go:356] "Generic (PLEG): container finished" podID="c45c31c9-7e31-44a6-b9ad-b09a78ee603c" containerID="9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71" exitCode=0
Jul 28 20:54:04.723462 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.723029    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"
Jul 28 20:54:04.723462 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.723065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw" event={"ID":"c45c31c9-7e31-44a6-b9ad-b09a78ee603c","Type":"ContainerDied","Data":"9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71"}
Jul 28 20:54:04.723462 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.723150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw" event={"ID":"c45c31c9-7e31-44a6-b9ad-b09a78ee603c","Type":"ContainerDied","Data":"1d3b650c2a39be9ea3c5bfefebc918457b54835fa8756affa667bf04b871a02a"}
Jul 28 20:54:04.723462 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.723186    2578 scope.go:117] "RemoveContainer" containerID="9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71"
Jul 28 20:54:04.724980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.724954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq" event={"ID":"fc2b74ec-2180-46c5-91d4-c23682a11a07","Type":"ContainerStarted","Data":"e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1"}
Jul 28 20:54:04.725148 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.725006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq" event={"ID":"fc2b74ec-2180-46c5-91d4-c23682a11a07","Type":"ContainerStarted","Data":"a81d93749dbe6b883ea2ae85b3b49f02c59b328895ae3860a49f392f84d7073f"}
Jul 28 20:54:04.725224 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.725177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 20:54:04.732456 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.732438    2578 scope.go:117] "RemoveContainer" containerID="9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71"
Jul 28 20:54:04.732721 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:54:04.732703    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71\": container with ID starting with 9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71 not found: ID does not exist" containerID="9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71"
Jul 28 20:54:04.732769 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.732731    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71"} err="failed to get container status \"9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71\": rpc error: code = NotFound desc = could not find container \"9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71\": container with ID starting with 9b419b52b793e9a1b8d3fbdefab0a0e687a397753d059368ebcdb07217a7be71 not found: ID does not exist"
Jul 28 20:54:04.791950 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.791878    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq" podStartSLOduration=1.791860314 podStartE2EDuration="1.791860314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:54:04.790778335 +0000 UTC m=+533.645622245" watchObservedRunningTime="2026-07-28 20:54:04.791860314 +0000 UTC m=+533.646704227"
Jul 28 20:54:04.819480 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.819451    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"]
Jul 28 20:54:04.820815 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:04.820789    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gnlnw"]
Jul 28 20:54:05.529790 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:05.529757    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c45c31c9-7e31-44a6-b9ad-b09a78ee603c" path="/var/lib/kubelet/pods/c45c31c9-7e31-44a6-b9ad-b09a78ee603c/volumes"
Jul 28 20:54:15.733057 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:15.733027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 20:54:15.792373 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:15.792335    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"]
Jul 28 20:54:15.792595 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:15.792556    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5" podUID="3f2208e9-f94e-4aef-a584-3b0cf75a904c" containerName="manager" containerID="cri-o://54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395" gracePeriod=10
Jul 28 20:54:16.032825 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.032800    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:54:16.118973 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.118943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f2208e9-f94e-4aef-a584-3b0cf75a904c-extensions-socket-volume\") pod \"3f2208e9-f94e-4aef-a584-3b0cf75a904c\" (UID: \"3f2208e9-f94e-4aef-a584-3b0cf75a904c\") "
Jul 28 20:54:16.119185 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.119037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slr92\" (UniqueName: \"kubernetes.io/projected/3f2208e9-f94e-4aef-a584-3b0cf75a904c-kube-api-access-slr92\") pod \"3f2208e9-f94e-4aef-a584-3b0cf75a904c\" (UID: \"3f2208e9-f94e-4aef-a584-3b0cf75a904c\") "
Jul 28 20:54:16.119332 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.119308    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2208e9-f94e-4aef-a584-3b0cf75a904c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3f2208e9-f94e-4aef-a584-3b0cf75a904c" (UID: "3f2208e9-f94e-4aef-a584-3b0cf75a904c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:54:16.121157 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.121131    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2208e9-f94e-4aef-a584-3b0cf75a904c-kube-api-access-slr92" (OuterVolumeSpecName: "kube-api-access-slr92") pod "3f2208e9-f94e-4aef-a584-3b0cf75a904c" (UID: "3f2208e9-f94e-4aef-a584-3b0cf75a904c"). InnerVolumeSpecName "kube-api-access-slr92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:54:16.220443 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.220409    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slr92\" (UniqueName: \"kubernetes.io/projected/3f2208e9-f94e-4aef-a584-3b0cf75a904c-kube-api-access-slr92\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:54:16.220443 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.220435    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f2208e9-f94e-4aef-a584-3b0cf75a904c-extensions-socket-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:54:16.778699 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.778665    2578 generic.go:356] "Generic (PLEG): container finished" podID="3f2208e9-f94e-4aef-a584-3b0cf75a904c" containerID="54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395" exitCode=0
Jul 28 20:54:16.779212 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.778712    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"
Jul 28 20:54:16.779212 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.778750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5" event={"ID":"3f2208e9-f94e-4aef-a584-3b0cf75a904c","Type":"ContainerDied","Data":"54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395"}
Jul 28 20:54:16.779212 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.778810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5" event={"ID":"3f2208e9-f94e-4aef-a584-3b0cf75a904c","Type":"ContainerDied","Data":"abd2033000d08191df1a711da9bf1006ab28b77cd1e042825452a5c863272626"}
Jul 28 20:54:16.779212 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.778833    2578 scope.go:117] "RemoveContainer" containerID="54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395"
Jul 28 20:54:16.787690 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.787675    2578 scope.go:117] "RemoveContainer" containerID="54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395"
Jul 28 20:54:16.787952 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:54:16.787935    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395\": container with ID starting with 54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395 not found: ID does not exist" containerID="54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395"
Jul 28 20:54:16.788012 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.787961    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395"} err="failed to get container status \"54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395\": rpc error: code = NotFound desc = could not find container \"54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395\": container with ID starting with 54bc95779aaa592913816ba27534b5b30e2d68db5bed1ff5f5c04d39b867f395 not found: ID does not exist"
Jul 28 20:54:16.799798 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.799771    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"]
Jul 28 20:54:16.806644 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:16.806623    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6svf5"]
Jul 28 20:54:17.528748 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:17.528708    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f2208e9-f94e-4aef-a584-3b0cf75a904c" path="/var/lib/kubelet/pods/3f2208e9-f94e-4aef-a584-3b0cf75a904c/volumes"
Jul 28 20:54:20.180609 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.180573    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"]
Jul 28 20:54:20.180963 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.180907    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2208e9-f94e-4aef-a584-3b0cf75a904c" containerName="manager"
Jul 28 20:54:20.180963 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.180919    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2208e9-f94e-4aef-a584-3b0cf75a904c" containerName="manager"
Jul 28 20:54:20.181033 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.180965    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2208e9-f94e-4aef-a584-3b0cf75a904c" containerName="manager"
Jul 28 20:54:20.185042 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.185022    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.188096 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.188062    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lrsqb\""
Jul 28 20:54:20.198330 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.198307    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"]
Jul 28 20:54:20.257125 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.257065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.257310 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.257137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.257310 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.257163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7de2788-2093-4a2b-bd7c-650e5ab13966-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.257310 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.257194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9nm\" (UniqueName: \"kubernetes.io/projected/b7de2788-2093-4a2b-bd7c-650e5ab13966-kube-api-access-vf9nm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.257310 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.257284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.257310 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.257310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.257565 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.257327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.257565 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.257372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.257565 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.257412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358292 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9nm\" (UniqueName: \"kubernetes.io/projected/b7de2788-2093-4a2b-bd7c-650e5ab13966-kube-api-access-vf9nm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358475 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358776 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358776 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358776 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7de2788-2093-4a2b-bd7c-650e5ab13966-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358924 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358924 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358853    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.358992 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.358933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.359193 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.359174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.359262 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.359248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7de2788-2093-4a2b-bd7c-650e5ab13966-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.361482 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.361461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.361745 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.361727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.371885 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.371855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9nm\" (UniqueName: \"kubernetes.io/projected/b7de2788-2093-4a2b-bd7c-650e5ab13966-kube-api-access-vf9nm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.375595 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.375572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7de2788-2093-4a2b-bd7c-650e5ab13966-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gv6ws\" (UID: \"b7de2788-2093-4a2b-bd7c-650e5ab13966\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.495071 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.495035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:20.632238 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.632213    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"]
Jul 28 20:54:20.633745 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:54:20.633720    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7de2788_2093_4a2b_bd7c_650e5ab13966.slice/crio-d48cc4e031385789d1216e04cf24c750c8d27a8cfebf6d5578f72aa751e73f6c WatchSource:0}: Error finding container d48cc4e031385789d1216e04cf24c750c8d27a8cfebf6d5578f72aa751e73f6c: Status 404 returned error can't find the container with id d48cc4e031385789d1216e04cf24c750c8d27a8cfebf6d5578f72aa751e73f6c
Jul 28 20:54:20.635718 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.635686    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 20:54:20.635808 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.635759    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 20:54:20.635868 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.635807    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 20:54:20.799464 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.799381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws" event={"ID":"b7de2788-2093-4a2b-bd7c-650e5ab13966","Type":"ContainerStarted","Data":"1d8883019d5e19b1bc2b17319e7eae76b205d0494161d6ffb9949d2223481476"}
Jul 28 20:54:20.799464 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.799444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws" event={"ID":"b7de2788-2093-4a2b-bd7c-650e5ab13966","Type":"ContainerStarted","Data":"d48cc4e031385789d1216e04cf24c750c8d27a8cfebf6d5578f72aa751e73f6c"}
Jul 28 20:54:20.820634 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:20.820581    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws" podStartSLOduration=0.820565058 podStartE2EDuration="820.565058ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:54:20.818680632 +0000 UTC m=+549.673524544" watchObservedRunningTime="2026-07-28 20:54:20.820565058 +0000 UTC m=+549.675408968"
Jul 28 20:54:21.495668 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:21.495635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:21.500121 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:21.500093    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:21.804364 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:21.804281    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:21.805403 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:21.805380    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:54:34.451333 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.451295    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b75hz"]
Jul 28 20:54:34.454943 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.454922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:34.457420 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.457400    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 20:54:34.457554 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.457494    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vzkpd\""
Jul 28 20:54:34.462844 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.462819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b75hz"]
Jul 28 20:54:34.547527 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.547498    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b75hz"]
Jul 28 20:54:34.579065 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.579029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b7c6139-12e1-43db-94fe-70e12de7b163-config-file\") pod \"limitador-limitador-64c94fd64f-b75hz\" (UID: \"5b7c6139-12e1-43db-94fe-70e12de7b163\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:34.579256 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.579146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75fsr\" (UniqueName: \"kubernetes.io/projected/5b7c6139-12e1-43db-94fe-70e12de7b163-kube-api-access-75fsr\") pod \"limitador-limitador-64c94fd64f-b75hz\" (UID: \"5b7c6139-12e1-43db-94fe-70e12de7b163\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:34.680656 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.680621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75fsr\" (UniqueName: \"kubernetes.io/projected/5b7c6139-12e1-43db-94fe-70e12de7b163-kube-api-access-75fsr\") pod \"limitador-limitador-64c94fd64f-b75hz\" (UID: \"5b7c6139-12e1-43db-94fe-70e12de7b163\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:34.680843 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.680724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b7c6139-12e1-43db-94fe-70e12de7b163-config-file\") pod \"limitador-limitador-64c94fd64f-b75hz\" (UID: \"5b7c6139-12e1-43db-94fe-70e12de7b163\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:34.681381 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.681361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b7c6139-12e1-43db-94fe-70e12de7b163-config-file\") pod \"limitador-limitador-64c94fd64f-b75hz\" (UID: \"5b7c6139-12e1-43db-94fe-70e12de7b163\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:34.688908 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.688887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75fsr\" (UniqueName: \"kubernetes.io/projected/5b7c6139-12e1-43db-94fe-70e12de7b163-kube-api-access-75fsr\") pod \"limitador-limitador-64c94fd64f-b75hz\" (UID: \"5b7c6139-12e1-43db-94fe-70e12de7b163\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:34.767247 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.767215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:34.902196 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:34.902173    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b75hz"]
Jul 28 20:54:34.904758 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:54:34.904733    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7c6139_12e1_43db_94fe_70e12de7b163.slice/crio-c2546dfc62ed3ff3b4f77127f73aee334c2ac8f67be0525d7fd1a1b2206c65e2 WatchSource:0}: Error finding container c2546dfc62ed3ff3b4f77127f73aee334c2ac8f67be0525d7fd1a1b2206c65e2: Status 404 returned error can't find the container with id c2546dfc62ed3ff3b4f77127f73aee334c2ac8f67be0525d7fd1a1b2206c65e2
Jul 28 20:54:35.874387 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:35.874286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz" event={"ID":"5b7c6139-12e1-43db-94fe-70e12de7b163","Type":"ContainerStarted","Data":"c2546dfc62ed3ff3b4f77127f73aee334c2ac8f67be0525d7fd1a1b2206c65e2"}
Jul 28 20:54:37.884911 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:37.884877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz" event={"ID":"5b7c6139-12e1-43db-94fe-70e12de7b163","Type":"ContainerStarted","Data":"41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6"}
Jul 28 20:54:37.885284 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:37.884976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:37.903631 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:37.903584    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz" podStartSLOduration=1.126735711 podStartE2EDuration="3.903571297s" totalImagesPullingTime="2.776835586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:54:37.901103186 +0000 UTC m=+566.755947095" watchObservedRunningTime="2026-07-28 20:54:37.903571297 +0000 UTC m=+566.758415210"
Jul 28 20:54:48.890488 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:48.890461    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:51.828805 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:51.828773    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b75hz"]
Jul 28 20:54:51.829246 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:51.828947    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz" podUID="5b7c6139-12e1-43db-94fe-70e12de7b163" containerName="limitador" containerID="cri-o://41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6" gracePeriod=30
Jul 28 20:54:52.368301 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.368277    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:52.544306 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.544275    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b7c6139-12e1-43db-94fe-70e12de7b163-config-file\") pod \"5b7c6139-12e1-43db-94fe-70e12de7b163\" (UID: \"5b7c6139-12e1-43db-94fe-70e12de7b163\") "
Jul 28 20:54:52.544472 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.544336    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75fsr\" (UniqueName: \"kubernetes.io/projected/5b7c6139-12e1-43db-94fe-70e12de7b163-kube-api-access-75fsr\") pod \"5b7c6139-12e1-43db-94fe-70e12de7b163\" (UID: \"5b7c6139-12e1-43db-94fe-70e12de7b163\") "
Jul 28 20:54:52.544614 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.544592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7c6139-12e1-43db-94fe-70e12de7b163-config-file" (OuterVolumeSpecName: "config-file") pod "5b7c6139-12e1-43db-94fe-70e12de7b163" (UID: "5b7c6139-12e1-43db-94fe-70e12de7b163"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:54:52.546497 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.546479    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7c6139-12e1-43db-94fe-70e12de7b163-kube-api-access-75fsr" (OuterVolumeSpecName: "kube-api-access-75fsr") pod "5b7c6139-12e1-43db-94fe-70e12de7b163" (UID: "5b7c6139-12e1-43db-94fe-70e12de7b163"). InnerVolumeSpecName "kube-api-access-75fsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:54:52.645663 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.645628    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b7c6139-12e1-43db-94fe-70e12de7b163-config-file\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:54:52.645663 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.645660    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75fsr\" (UniqueName: \"kubernetes.io/projected/5b7c6139-12e1-43db-94fe-70e12de7b163-kube-api-access-75fsr\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:54:52.956602 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.956517    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b7c6139-12e1-43db-94fe-70e12de7b163" containerID="41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6" exitCode=0
Jul 28 20:54:52.956602 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.956566    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz"
Jul 28 20:54:52.957029 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.956594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz" event={"ID":"5b7c6139-12e1-43db-94fe-70e12de7b163","Type":"ContainerDied","Data":"41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6"}
Jul 28 20:54:52.957029 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.956650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-b75hz" event={"ID":"5b7c6139-12e1-43db-94fe-70e12de7b163","Type":"ContainerDied","Data":"c2546dfc62ed3ff3b4f77127f73aee334c2ac8f67be0525d7fd1a1b2206c65e2"}
Jul 28 20:54:52.957029 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.956685    2578 scope.go:117] "RemoveContainer" containerID="41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6"
Jul 28 20:54:52.965281 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.965265    2578 scope.go:117] "RemoveContainer" containerID="41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6"
Jul 28 20:54:52.965526 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:54:52.965508    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6\": container with ID starting with 41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6 not found: ID does not exist" containerID="41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6"
Jul 28 20:54:52.965594 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.965539    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6"} err="failed to get container status \"41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6\": rpc error: code = NotFound desc = could not find container \"41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6\": container with ID starting with 41375d50b42032b590c46e61c4ff37acccd47c4cf989c356cf5e2a77b70ecec6 not found: ID does not exist"
Jul 28 20:54:52.976994 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.976972    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b75hz"]
Jul 28 20:54:52.980374 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:52.980356    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-b75hz"]
Jul 28 20:54:53.529427 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:53.529397    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b7c6139-12e1-43db-94fe-70e12de7b163" path="/var/lib/kubelet/pods/5b7c6139-12e1-43db-94fe-70e12de7b163/volumes"
Jul 28 20:54:54.108429 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.108398    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"]
Jul 28 20:54:54.108888 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.108871    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b7c6139-12e1-43db-94fe-70e12de7b163" containerName="limitador"
Jul 28 20:54:54.108955 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.108891    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b7c6139-12e1-43db-94fe-70e12de7b163" containerName="limitador"
Jul 28 20:54:54.109006 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.108982    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b7c6139-12e1-43db-94fe-70e12de7b163" containerName="limitador"
Jul 28 20:54:54.113428 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.113408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"
Jul 28 20:54:54.115911 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.115877    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:54:54.115911 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.115893    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:54:54.116095 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.115955    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 28 20:54:54.117001 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.116982    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-z4jbs\""
Jul 28 20:54:54.118997 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.118980    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"]
Jul 28 20:54:54.158360 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.158334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz69\" (UniqueName: \"kubernetes.io/projected/f416da26-7565-44fc-9ddd-1669731d2be2-kube-api-access-5zz69\") pod \"postgres-69fb4594cf-hmnsl\" (UID: \"f416da26-7565-44fc-9ddd-1669731d2be2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"
Jul 28 20:54:54.158462 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.158370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f416da26-7565-44fc-9ddd-1669731d2be2-data\") pod \"postgres-69fb4594cf-hmnsl\" (UID: \"f416da26-7565-44fc-9ddd-1669731d2be2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"
Jul 28 20:54:54.259058 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.259027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz69\" (UniqueName: \"kubernetes.io/projected/f416da26-7565-44fc-9ddd-1669731d2be2-kube-api-access-5zz69\") pod \"postgres-69fb4594cf-hmnsl\" (UID: \"f416da26-7565-44fc-9ddd-1669731d2be2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"
Jul 28 20:54:54.259244 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.259070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f416da26-7565-44fc-9ddd-1669731d2be2-data\") pod \"postgres-69fb4594cf-hmnsl\" (UID: \"f416da26-7565-44fc-9ddd-1669731d2be2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"
Jul 28 20:54:54.259480 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.259464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f416da26-7565-44fc-9ddd-1669731d2be2-data\") pod \"postgres-69fb4594cf-hmnsl\" (UID: \"f416da26-7565-44fc-9ddd-1669731d2be2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"
Jul 28 20:54:54.267312 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.267294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz69\" (UniqueName: \"kubernetes.io/projected/f416da26-7565-44fc-9ddd-1669731d2be2-kube-api-access-5zz69\") pod \"postgres-69fb4594cf-hmnsl\" (UID: \"f416da26-7565-44fc-9ddd-1669731d2be2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"
Jul 28 20:54:54.425427 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.425350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"
Jul 28 20:54:54.559401 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.559370    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"]
Jul 28 20:54:54.562945 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:54:54.562915    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf416da26_7565_44fc_9ddd_1669731d2be2.slice/crio-8aae6a6cf68f96ea9380ef11b9cc960e9611be555ef6eca83cf33be417551c19 WatchSource:0}: Error finding container 8aae6a6cf68f96ea9380ef11b9cc960e9611be555ef6eca83cf33be417551c19: Status 404 returned error can't find the container with id 8aae6a6cf68f96ea9380ef11b9cc960e9611be555ef6eca83cf33be417551c19
Jul 28 20:54:54.966858 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:54:54.966824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl" event={"ID":"f416da26-7565-44fc-9ddd-1669731d2be2","Type":"ContainerStarted","Data":"8aae6a6cf68f96ea9380ef11b9cc960e9611be555ef6eca83cf33be417551c19"}
Jul 28 20:55:00.996469 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:00.996426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl" event={"ID":"f416da26-7565-44fc-9ddd-1669731d2be2","Type":"ContainerStarted","Data":"8a6d2debd47ae579fa44fd4e51f9d4c5696a651d8339136bf7995ad0997751b7"}
Jul 28 20:55:00.996852 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:00.996551    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"
Jul 28 20:55:01.012382 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:01.012330    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl" podStartSLOduration=1.260391969 podStartE2EDuration="7.012311545s" totalImagesPullingTime="5.751919576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:01.011538923 +0000 UTC m=+589.866382836" watchObservedRunningTime="2026-07-28 20:55:01.012311545 +0000 UTC m=+589.867155455"
Jul 28 20:55:07.028939 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:07.028912    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hmnsl"
Jul 28 20:55:15.254506 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.254474    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86c5b47454-lxmc6"]
Jul 28 20:55:15.278289 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.278257    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86c5b47454-lxmc6"]
Jul 28 20:55:15.278477 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.278379    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:15.280989 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.280966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 20:55:15.280989 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.280985    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8q6cc\""
Jul 28 20:55:15.281244 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.281006    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 20:55:15.338648 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.338617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4n5\" (UniqueName: \"kubernetes.io/projected/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-kube-api-access-tn4n5\") pod \"maas-controller-86c5b47454-lxmc6\" (UID: \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\") " pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:15.338817 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.338666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-webhook-cert\") pod \"maas-controller-86c5b47454-lxmc6\" (UID: \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\") " pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:15.439775 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.439733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4n5\" (UniqueName: \"kubernetes.io/projected/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-kube-api-access-tn4n5\") pod \"maas-controller-86c5b47454-lxmc6\" (UID: \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\") " pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:15.439945 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.439786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-webhook-cert\") pod \"maas-controller-86c5b47454-lxmc6\" (UID: \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\") " pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:15.439945 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:55:15.439910    2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 28 20:55:15.440028 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:55:15.439967    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-webhook-cert podName:0180fd6a-b41f-4c96-b03f-ba79bf424e1d nodeName:}" failed. No retries permitted until 2026-07-28 20:55:15.939951456 +0000 UTC m=+604.794795345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-webhook-cert") pod "maas-controller-86c5b47454-lxmc6" (UID: "0180fd6a-b41f-4c96-b03f-ba79bf424e1d") : secret "maas-controller-webhook-cert" not found
Jul 28 20:55:15.444014 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.443990    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76686b4476-579wv"]
Jul 28 20:55:15.447487 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.447467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:15.451725 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.451706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4n5\" (UniqueName: \"kubernetes.io/projected/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-kube-api-access-tn4n5\") pod \"maas-controller-86c5b47454-lxmc6\" (UID: \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\") " pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:15.456457 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.456436    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76686b4476-579wv"]
Jul 28 20:55:15.540623 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.540542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcsg9\" (UniqueName: \"kubernetes.io/projected/dfa4a79b-32ff-457d-bc81-622ee6d5140b-kube-api-access-bcsg9\") pod \"maas-controller-76686b4476-579wv\" (UID: \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\") " pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:15.540766 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.540656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfa4a79b-32ff-457d-bc81-622ee6d5140b-webhook-cert\") pod \"maas-controller-76686b4476-579wv\" (UID: \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\") " pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:15.641952 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.641915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfa4a79b-32ff-457d-bc81-622ee6d5140b-webhook-cert\") pod \"maas-controller-76686b4476-579wv\" (UID: \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\") " pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:15.642142 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.641979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsg9\" (UniqueName: \"kubernetes.io/projected/dfa4a79b-32ff-457d-bc81-622ee6d5140b-kube-api-access-bcsg9\") pod \"maas-controller-76686b4476-579wv\" (UID: \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\") " pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:15.642142 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:55:15.642073    2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 28 20:55:15.642223 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:55:15.642174    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfa4a79b-32ff-457d-bc81-622ee6d5140b-webhook-cert podName:dfa4a79b-32ff-457d-bc81-622ee6d5140b nodeName:}" failed. No retries permitted until 2026-07-28 20:55:16.142156575 +0000 UTC m=+604.997000466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/dfa4a79b-32ff-457d-bc81-622ee6d5140b-webhook-cert") pod "maas-controller-76686b4476-579wv" (UID: "dfa4a79b-32ff-457d-bc81-622ee6d5140b") : secret "maas-controller-webhook-cert" not found
Jul 28 20:55:15.651559 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.651537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsg9\" (UniqueName: \"kubernetes.io/projected/dfa4a79b-32ff-457d-bc81-622ee6d5140b-kube-api-access-bcsg9\") pod \"maas-controller-76686b4476-579wv\" (UID: \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\") " pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:15.944632 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.944552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-webhook-cert\") pod \"maas-controller-86c5b47454-lxmc6\" (UID: \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\") " pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:15.947048 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:15.947029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-webhook-cert\") pod \"maas-controller-86c5b47454-lxmc6\" (UID: \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\") " pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:16.146879 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:16.146849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfa4a79b-32ff-457d-bc81-622ee6d5140b-webhook-cert\") pod \"maas-controller-76686b4476-579wv\" (UID: \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\") " pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:16.149321 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:16.149300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfa4a79b-32ff-457d-bc81-622ee6d5140b-webhook-cert\") pod \"maas-controller-76686b4476-579wv\" (UID: \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\") " pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:16.190239 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:16.190209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:16.319130 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:16.319105    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86c5b47454-lxmc6"]
Jul 28 20:55:16.320726 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:55:16.320699    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0180fd6a_b41f_4c96_b03f_ba79bf424e1d.slice/crio-19c20d0620c5db13cfa1f4f987845f595ede6676cd556b454663ca8bd2355794 WatchSource:0}: Error finding container 19c20d0620c5db13cfa1f4f987845f595ede6676cd556b454663ca8bd2355794: Status 404 returned error can't find the container with id 19c20d0620c5db13cfa1f4f987845f595ede6676cd556b454663ca8bd2355794
Jul 28 20:55:16.365171 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:16.365149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:16.492767 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:16.492745    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76686b4476-579wv"]
Jul 28 20:55:16.494629 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:55:16.494600    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa4a79b_32ff_457d_bc81_622ee6d5140b.slice/crio-f27fd1dd4b287f943e04c7ccc82c5c0060b79ad7ef14b862b2e2824693a4abac WatchSource:0}: Error finding container f27fd1dd4b287f943e04c7ccc82c5c0060b79ad7ef14b862b2e2824693a4abac: Status 404 returned error can't find the container with id f27fd1dd4b287f943e04c7ccc82c5c0060b79ad7ef14b862b2e2824693a4abac
Jul 28 20:55:17.066254 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:17.066216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76686b4476-579wv" event={"ID":"dfa4a79b-32ff-457d-bc81-622ee6d5140b","Type":"ContainerStarted","Data":"f27fd1dd4b287f943e04c7ccc82c5c0060b79ad7ef14b862b2e2824693a4abac"}
Jul 28 20:55:17.067318 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:17.067286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c5b47454-lxmc6" event={"ID":"0180fd6a-b41f-4c96-b03f-ba79bf424e1d","Type":"ContainerStarted","Data":"19c20d0620c5db13cfa1f4f987845f595ede6676cd556b454663ca8bd2355794"}
Jul 28 20:55:20.087337 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:20.087293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c5b47454-lxmc6" event={"ID":"0180fd6a-b41f-4c96-b03f-ba79bf424e1d","Type":"ContainerStarted","Data":"9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8"}
Jul 28 20:55:20.087840 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:20.087383    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:20.089029 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:20.089004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76686b4476-579wv" event={"ID":"dfa4a79b-32ff-457d-bc81-622ee6d5140b","Type":"ContainerStarted","Data":"3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803"}
Jul 28 20:55:20.089173 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:20.089096    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:20.130984 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:20.130921    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86c5b47454-lxmc6" podStartSLOduration=1.922312939 podStartE2EDuration="5.130820784s" totalImagesPullingTime="3.208507845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:20.130046922 +0000 UTC m=+608.984890837" watchObservedRunningTime="2026-07-28 20:55:20.130820784 +0000 UTC m=+608.985664697"
Jul 28 20:55:20.163627 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:20.163575    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76686b4476-579wv" podStartSLOduration=2.12244892 podStartE2EDuration="5.163558616s" totalImagesPullingTime="3.041109696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:20.162978442 +0000 UTC m=+609.017822388" watchObservedRunningTime="2026-07-28 20:55:20.163558616 +0000 UTC m=+609.018402527"
Jul 28 20:55:31.096229 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.096195    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:31.096660 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.096277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:55:31.153743 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.153704    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86c5b47454-lxmc6"]
Jul 28 20:55:31.153970 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.153947    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86c5b47454-lxmc6" podUID="0180fd6a-b41f-4c96-b03f-ba79bf424e1d" containerName="manager" containerID="cri-o://9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8" gracePeriod=30
Jul 28 20:55:31.400365 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.400342    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:31.487980 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.487949    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-webhook-cert\") pod \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\" (UID: \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\") "
Jul 28 20:55:31.488149 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.488064    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn4n5\" (UniqueName: \"kubernetes.io/projected/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-kube-api-access-tn4n5\") pod \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\" (UID: \"0180fd6a-b41f-4c96-b03f-ba79bf424e1d\") "
Jul 28 20:55:31.490257 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.490224    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-kube-api-access-tn4n5" (OuterVolumeSpecName: "kube-api-access-tn4n5") pod "0180fd6a-b41f-4c96-b03f-ba79bf424e1d" (UID: "0180fd6a-b41f-4c96-b03f-ba79bf424e1d"). InnerVolumeSpecName "kube-api-access-tn4n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:55:31.490257 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.490244    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0180fd6a-b41f-4c96-b03f-ba79bf424e1d" (UID: "0180fd6a-b41f-4c96-b03f-ba79bf424e1d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:55:31.589450 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.589427    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4n5\" (UniqueName: \"kubernetes.io/projected/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-kube-api-access-tn4n5\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:55:31.589450 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:31.589448    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0180fd6a-b41f-4c96-b03f-ba79bf424e1d-webhook-cert\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:55:32.140752 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:32.140716    2578 generic.go:356] "Generic (PLEG): container finished" podID="0180fd6a-b41f-4c96-b03f-ba79bf424e1d" containerID="9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8" exitCode=0
Jul 28 20:55:32.141160 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:32.140767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c5b47454-lxmc6" event={"ID":"0180fd6a-b41f-4c96-b03f-ba79bf424e1d","Type":"ContainerDied","Data":"9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8"}
Jul 28 20:55:32.141160 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:32.140773    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c5b47454-lxmc6"
Jul 28 20:55:32.141160 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:32.140817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c5b47454-lxmc6" event={"ID":"0180fd6a-b41f-4c96-b03f-ba79bf424e1d","Type":"ContainerDied","Data":"19c20d0620c5db13cfa1f4f987845f595ede6676cd556b454663ca8bd2355794"}
Jul 28 20:55:32.141160 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:32.140847    2578 scope.go:117] "RemoveContainer" containerID="9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8"
Jul 28 20:55:32.148910 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:32.148893    2578 scope.go:117] "RemoveContainer" containerID="9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8"
Jul 28 20:55:32.149190 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:55:32.149172    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8\": container with ID starting with 9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8 not found: ID does not exist" containerID="9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8"
Jul 28 20:55:32.149250 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:32.149201    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8"} err="failed to get container status \"9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8\": rpc error: code = NotFound desc = could not find container \"9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8\": container with ID starting with 9e9195bdbe3c15ce7ebea858283f5a4bac694076f5e44faaad9e427f787fdbb8 not found: ID does not exist"
Jul 28 20:55:32.157308 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:32.157286    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86c5b47454-lxmc6"]
Jul 28 20:55:32.162599 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:32.162580    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86c5b47454-lxmc6"]
Jul 28 20:55:33.529498 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:33.529425    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0180fd6a-b41f-4c96-b03f-ba79bf424e1d" path="/var/lib/kubelet/pods/0180fd6a-b41f-4c96-b03f-ba79bf424e1d/volumes"
Jul 28 20:55:52.007832 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.007787    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-76zbd"]
Jul 28 20:55:52.008420 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.008145    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0180fd6a-b41f-4c96-b03f-ba79bf424e1d" containerName="manager"
Jul 28 20:55:52.008420 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.008158    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0180fd6a-b41f-4c96-b03f-ba79bf424e1d" containerName="manager"
Jul 28 20:55:52.008420 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.008221    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0180fd6a-b41f-4c96-b03f-ba79bf424e1d" containerName="manager"
Jul 28 20:55:52.014886 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.014866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd"
Jul 28 20:55:52.017778 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.017713    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ls7ns\""
Jul 28 20:55:52.018011 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.017745    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 28 20:55:52.024109 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.024065    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-76zbd"]
Jul 28 20:55:52.055961 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.055924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fe5e12-e5b9-4359-8a5b-342ff454649e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-76zbd\" (UID: \"73fe5e12-e5b9-4359-8a5b-342ff454649e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd"
Jul 28 20:55:52.056136 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.055996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4ng2\" (UniqueName: \"kubernetes.io/projected/73fe5e12-e5b9-4359-8a5b-342ff454649e-kube-api-access-l4ng2\") pod \"payload-processing-5cc6cd444b-76zbd\" (UID: \"73fe5e12-e5b9-4359-8a5b-342ff454649e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd"
Jul 28 20:55:52.156484 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.156455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fe5e12-e5b9-4359-8a5b-342ff454649e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-76zbd\" (UID: \"73fe5e12-e5b9-4359-8a5b-342ff454649e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd"
Jul 28 20:55:52.156655 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.156498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ng2\" (UniqueName: \"kubernetes.io/projected/73fe5e12-e5b9-4359-8a5b-342ff454649e-kube-api-access-l4ng2\") pod \"payload-processing-5cc6cd444b-76zbd\" (UID: \"73fe5e12-e5b9-4359-8a5b-342ff454649e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd"
Jul 28 20:55:52.157418 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.157394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fe5e12-e5b9-4359-8a5b-342ff454649e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-76zbd\" (UID: \"73fe5e12-e5b9-4359-8a5b-342ff454649e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd"
Jul 28 20:55:52.171762 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.171733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ng2\" (UniqueName: \"kubernetes.io/projected/73fe5e12-e5b9-4359-8a5b-342ff454649e-kube-api-access-l4ng2\") pod \"payload-processing-5cc6cd444b-76zbd\" (UID: \"73fe5e12-e5b9-4359-8a5b-342ff454649e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd"
Jul 28 20:55:52.327313 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.327231    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd"
Jul 28 20:55:52.459151 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:52.459129    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-76zbd"]
Jul 28 20:55:52.461747 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:55:52.461715    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fe5e12_e5b9_4359_8a5b_342ff454649e.slice/crio-7a7a64e5a5e6e7a1ff5d07b47e09e66fc195bbbb63675a4496e7378028f013e2 WatchSource:0}: Error finding container 7a7a64e5a5e6e7a1ff5d07b47e09e66fc195bbbb63675a4496e7378028f013e2: Status 404 returned error can't find the container with id 7a7a64e5a5e6e7a1ff5d07b47e09e66fc195bbbb63675a4496e7378028f013e2
Jul 28 20:55:53.239355 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:53.239305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd" event={"ID":"73fe5e12-e5b9-4359-8a5b-342ff454649e","Type":"ContainerStarted","Data":"7a7a64e5a5e6e7a1ff5d07b47e09e66fc195bbbb63675a4496e7378028f013e2"}
Jul 28 20:55:56.254258 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:56.254219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd" event={"ID":"73fe5e12-e5b9-4359-8a5b-342ff454649e","Type":"ContainerStarted","Data":"6300f19fe2d9cfdf32d758188f4c026a30b84a7aefb17449de64dc154007446e"}
Jul 28 20:55:56.256582 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:56.254359    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd"
Jul 28 20:55:56.271396 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:56.271351    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd" podStartSLOduration=2.564379243 podStartE2EDuration="5.271337419s" totalImagesPullingTime="2.706958176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:56.26993584 +0000 UTC m=+645.124779767" watchObservedRunningTime="2026-07-28 20:55:56.271337419 +0000 UTC m=+645.126181329"
Jul 28 20:55:59.269008 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:59.268970    2578 generic.go:356] "Generic (PLEG): container finished" podID="b7de2788-2093-4a2b-bd7c-650e5ab13966" containerID="1d8883019d5e19b1bc2b17319e7eae76b205d0494161d6ffb9949d2223481476" exitCode=0
Jul 28 20:55:59.269534 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:59.269035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws" event={"ID":"b7de2788-2093-4a2b-bd7c-650e5ab13966","Type":"ContainerDied","Data":"1d8883019d5e19b1bc2b17319e7eae76b205d0494161d6ffb9949d2223481476"}
Jul 28 20:55:59.269599 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:59.269551    2578 scope.go:117] "RemoveContainer" containerID="1d8883019d5e19b1bc2b17319e7eae76b205d0494161d6ffb9949d2223481476"
Jul 28 20:55:59.270585 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:59.270551    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 20:55:59.270696 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:59.270623    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 20:55:59.270696 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:55:59.270670    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 20:56:00.275295 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:00.275263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws" event={"ID":"b7de2788-2093-4a2b-bd7c-650e5ab13966","Type":"ContainerStarted","Data":"ab908f4f888ce2e0328e28ee858cb48b6e05d34025a02d0a07734e003531b1de"}
Jul 28 20:56:00.495981 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:00.495949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:56:00.496337 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:00.496310    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gv6ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Jul 28 20:56:00.496502 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:00.496469    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws" podUID="b7de2788-2093-4a2b-bd7c-650e5ab13966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Jul 28 20:56:01.496463 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:01.496418    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gv6ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Jul 28 20:56:01.496954 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:01.496534    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws" podUID="b7de2788-2093-4a2b-bd7c-650e5ab13966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Jul 28 20:56:02.496377 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:02.496335    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gv6ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Jul 28 20:56:02.496698 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:02.496665    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws" podUID="b7de2788-2093-4a2b-bd7c-650e5ab13966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Jul 28 20:56:03.495956 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:03.495911    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gv6ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Jul 28 20:56:03.496316 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:03.496280    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws" podUID="b7de2788-2093-4a2b-bd7c-650e5ab13966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Jul 28 20:56:04.500348 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:04.500304    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:56:04.500898 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:04.500669    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:56:04.501383 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:04.501362    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gv6ws"
Jul 28 20:56:07.261412 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:07.261385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-76zbd"
Jul 28 20:56:33.637629 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.637597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"]
Jul 28 20:56:33.645724 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.645696    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.648370 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.648347    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hkgj4\""
Jul 28 20:56:33.648604 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.648395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 28 20:56:33.648685 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.648401    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 20:56:33.648742 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.648418    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 20:56:33.649503 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.649481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"]
Jul 28 20:56:33.741278 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.741246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.741443 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.741320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.741443 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.741351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c6f3fd-9c63-467e-928a-e15ecb6ec840-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.741443 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.741367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vtc2\" (UniqueName: \"kubernetes.io/projected/25c6f3fd-9c63-467e-928a-e15ecb6ec840-kube-api-access-9vtc2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.741443 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.741394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.741443 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.741436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.842360 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.842327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.842521 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.842410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.842521 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.842441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c6f3fd-9c63-467e-928a-e15ecb6ec840-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.842521 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.842457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vtc2\" (UniqueName: \"kubernetes.io/projected/25c6f3fd-9c63-467e-928a-e15ecb6ec840-kube-api-access-9vtc2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.842521 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.842480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.842521 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.842518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.842823 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.842797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.842880 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.842850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.842880 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.842865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.844887 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.844867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c6f3fd-9c63-467e-928a-e15ecb6ec840-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.845125 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.845109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c6f3fd-9c63-467e-928a-e15ecb6ec840-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.851235 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.851218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vtc2\" (UniqueName: \"kubernetes.io/projected/25c6f3fd-9c63-467e-928a-e15ecb6ec840-kube-api-access-9vtc2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qxsm2\" (UID: \"25c6f3fd-9c63-467e-928a-e15ecb6ec840\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:33.956850 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:33.956771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:34.092280 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:34.092257    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"]
Jul 28 20:56:34.094896 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:56:34.094870    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c6f3fd_9c63_467e_928a_e15ecb6ec840.slice/crio-9bfc73d694dd8abf65891751609abcca208683452fbc1fdcfb9eee98f50213a1 WatchSource:0}: Error finding container 9bfc73d694dd8abf65891751609abcca208683452fbc1fdcfb9eee98f50213a1: Status 404 returned error can't find the container with id 9bfc73d694dd8abf65891751609abcca208683452fbc1fdcfb9eee98f50213a1
Jul 28 20:56:34.096142 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:34.096123    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:56:34.432483 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:34.432437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2" event={"ID":"25c6f3fd-9c63-467e-928a-e15ecb6ec840","Type":"ContainerStarted","Data":"9bfc73d694dd8abf65891751609abcca208683452fbc1fdcfb9eee98f50213a1"}
Jul 28 20:56:35.437986 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:35.437904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2" event={"ID":"25c6f3fd-9c63-467e-928a-e15ecb6ec840","Type":"ContainerStarted","Data":"354cdc5ac97e2663e550f233bca1c13187397b60422026d64af168dfd64d96bd"}
Jul 28 20:56:43.957560 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:43.957530    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:43.957955 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:43.957592    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:43.969845 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:43.969823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:43.988028 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:43.987983    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2" podStartSLOduration=10.006973215 podStartE2EDuration="10.987969051s" totalImagesPullingTime="980.995836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:56:35.480699867 +0000 UTC m=+684.335543778" watchObservedRunningTime="2026-07-28 20:56:43.987969051 +0000 UTC m=+692.842812961"
Jul 28 20:56:44.489718 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:44.489686    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qxsm2"
Jul 28 20:56:53.127756 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.127722    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"]
Jul 28 20:56:53.132026 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.132003    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.134302 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.134280    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 28 20:56:53.141483 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.141462    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"]
Jul 28 20:56:53.210580 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.210550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.210742 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.210603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.210742 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.210625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djx5f\" (UniqueName: \"kubernetes.io/projected/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-kube-api-access-djx5f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.210742 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.210715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.210914 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.210775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.210914 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.210815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.311464 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.311414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.311464 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.311451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djx5f\" (UniqueName: \"kubernetes.io/projected/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-kube-api-access-djx5f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.311752 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.311490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.311752 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.311531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.311752 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.311559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.311752 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.311667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.312062 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.312033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.312062 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.312048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.312331 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.312315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.314435 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.314413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.314816 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.314784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.319002 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.318983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djx5f\" (UniqueName: \"kubernetes.io/projected/e8fe03cc-ec6a-49ca-a7ca-68220d572a1b-kube-api-access-djx5f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg\" (UID: \"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.443812 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.443728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:56:53.579319 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:53.579292    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"]
Jul 28 20:56:53.581520 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:56:53.581490    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fe03cc_ec6a_49ca_a7ca_68220d572a1b.slice/crio-41f77c182fd5a825b521a4024bc8ef26bb4acb219c41e6a79e8164362d9221a9 WatchSource:0}: Error finding container 41f77c182fd5a825b521a4024bc8ef26bb4acb219c41e6a79e8164362d9221a9: Status 404 returned error can't find the container with id 41f77c182fd5a825b521a4024bc8ef26bb4acb219c41e6a79e8164362d9221a9
Jul 28 20:56:54.526525 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:54.526489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg" event={"ID":"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b","Type":"ContainerStarted","Data":"e4dc0a003c74d23cec2bdd78a559024ebc5b49fc0c021b8051c879b0d4b77666"}
Jul 28 20:56:54.526905 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:54.526548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg" event={"ID":"e8fe03cc-ec6a-49ca-a7ca-68220d572a1b","Type":"ContainerStarted","Data":"41f77c182fd5a825b521a4024bc8ef26bb4acb219c41e6a79e8164362d9221a9"}
Jul 28 20:56:54.545799 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:56:54.545758    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg" podStartSLOduration=1.373397927 podStartE2EDuration="1.545742634s" totalImagesPullingTime="172.344707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:56:54.543656405 +0000 UTC m=+703.398500352" watchObservedRunningTime="2026-07-28 20:56:54.545742634 +0000 UTC m=+703.400586545"
Jul 28 20:57:02.539430 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.539352    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"]
Jul 28 20:57:02.543005 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.542987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.545436 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.545419    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 28 20:57:02.551845 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.551826    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"]
Jul 28 20:57:02.699199 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.699158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj28s\" (UniqueName: \"kubernetes.io/projected/60fa645a-e0e2-484d-b636-6220db27ab89-kube-api-access-vj28s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.699391 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.699242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fa645a-e0e2-484d-b636-6220db27ab89-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.699391 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.699367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.699491 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.699421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.699491 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.699447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.699590 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.699510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.800706 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.800621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.800706 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.800664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.800706 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.800681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.800958 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.800801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.800958 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.800850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj28s\" (UniqueName: \"kubernetes.io/projected/60fa645a-e0e2-484d-b636-6220db27ab89-kube-api-access-vj28s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.800958 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.800898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fa645a-e0e2-484d-b636-6220db27ab89-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.801143 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.801008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.801143 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.801067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.801329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.801232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.803165 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.803144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fa645a-e0e2-484d-b636-6220db27ab89-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.803395 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.803378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fa645a-e0e2-484d-b636-6220db27ab89-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.809821 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.809797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj28s\" (UniqueName: \"kubernetes.io/projected/60fa645a-e0e2-484d-b636-6220db27ab89-kube-api-access-vj28s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pc4bc\" (UID: \"60fa645a-e0e2-484d-b636-6220db27ab89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.854952 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.854917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:02.998333 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:02.998307    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"]
Jul 28 20:57:02.999942 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:57:02.999915    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60fa645a_e0e2_484d_b636_6220db27ab89.slice/crio-0de010ac3bd43a97432c84317b93d2a251de61a8bdbe939d17d569413e0c984e WatchSource:0}: Error finding container 0de010ac3bd43a97432c84317b93d2a251de61a8bdbe939d17d569413e0c984e: Status 404 returned error can't find the container with id 0de010ac3bd43a97432c84317b93d2a251de61a8bdbe939d17d569413e0c984e
Jul 28 20:57:03.444228 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:03.444144    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:57:03.444394 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:03.444254    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:57:03.456811 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:03.456783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:57:03.575178 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:03.575139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc" event={"ID":"60fa645a-e0e2-484d-b636-6220db27ab89","Type":"ContainerStarted","Data":"a862fdb43770eb0d26567fe6c25f9f5e5b4282443177c241924da82cb6235c40"}
Jul 28 20:57:03.575672 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:03.575195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc" event={"ID":"60fa645a-e0e2-484d-b636-6220db27ab89","Type":"ContainerStarted","Data":"0de010ac3bd43a97432c84317b93d2a251de61a8bdbe939d17d569413e0c984e"}
Jul 28 20:57:03.586943 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:03.586910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg"
Jul 28 20:57:03.594337 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:03.594290    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc" podStartSLOduration=1.423053017 podStartE2EDuration="1.594274333s" totalImagesPullingTime="171.221316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:57:03.593942 +0000 UTC m=+712.448785914" watchObservedRunningTime="2026-07-28 20:57:03.594274333 +0000 UTC m=+712.449118265"
Jul 28 20:57:06.834734 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:06.834703    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"]
Jul 28 20:57:06.839158 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:06.839134    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:06.841506 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:06.841482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 28 20:57:06.850804 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:06.850772    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"]
Jul 28 20:57:06.936147 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:06.936107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:06.936334 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:06.936178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84114904-8707-48e5-85cb-8f576e6e690b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:06.936334 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:06.936206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:06.936334 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:06.936226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:06.936450 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:06.936327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:06.936450 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:06.936414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6p7\" (UniqueName: \"kubernetes.io/projected/84114904-8707-48e5-85cb-8f576e6e690b-kube-api-access-7w6p7\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.037032 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.036999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6p7\" (UniqueName: \"kubernetes.io/projected/84114904-8707-48e5-85cb-8f576e6e690b-kube-api-access-7w6p7\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.037245 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.037057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.037245 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.037122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84114904-8707-48e5-85cb-8f576e6e690b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.037245 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.037148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.037245 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.037164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.037245 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.037195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.037544 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.037522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.037604 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.037579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.037663 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.037606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.039398 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.039373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84114904-8707-48e5-85cb-8f576e6e690b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.039568 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.039548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84114904-8707-48e5-85cb-8f576e6e690b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.044883 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.044857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6p7\" (UniqueName: \"kubernetes.io/projected/84114904-8707-48e5-85cb-8f576e6e690b-kube-api-access-7w6p7\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgq59\" (UID: \"84114904-8707-48e5-85cb-8f576e6e690b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.148122 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.148003    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:07.286959 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.286934    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"]
Jul 28 20:57:07.288857 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:57:07.288820    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84114904_8707_48e5_85cb_8f576e6e690b.slice/crio-4cde8abff62d6664287fcccf842b683411fc2abcde5fba50634d83e835f37d57 WatchSource:0}: Error finding container 4cde8abff62d6664287fcccf842b683411fc2abcde5fba50634d83e835f37d57: Status 404 returned error can't find the container with id 4cde8abff62d6664287fcccf842b683411fc2abcde5fba50634d83e835f37d57
Jul 28 20:57:07.595189 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.595154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59" event={"ID":"84114904-8707-48e5-85cb-8f576e6e690b","Type":"ContainerStarted","Data":"c7b364215976c44330f9019151be0ef8726358c4aaa0c586614fb8e3babf6e30"}
Jul 28 20:57:07.595359 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.595198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59" event={"ID":"84114904-8707-48e5-85cb-8f576e6e690b","Type":"ContainerStarted","Data":"4cde8abff62d6664287fcccf842b683411fc2abcde5fba50634d83e835f37d57"}
Jul 28 20:57:07.612329 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:07.612273    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59" podStartSLOduration=1.413891896 podStartE2EDuration="1.612258498s" totalImagesPullingTime="198.366602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:57:07.610890717 +0000 UTC m=+716.465734656" watchObservedRunningTime="2026-07-28 20:57:07.612258498 +0000 UTC m=+716.467102445"
Jul 28 20:57:12.855658 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:12.855614    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:12.856149 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:12.855720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:12.867990 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:12.867964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:13.633491 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:13.633463    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pc4bc"
Jul 28 20:57:17.148856 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:17.148818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:17.149344 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:17.148870    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:17.161799 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:17.161777    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:57:17.651847 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:57:17.651814    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgq59"
Jul 28 20:58:36.705490 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:36.705399    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76686b4476-579wv"]
Jul 28 20:58:36.706036 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:36.705615    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-76686b4476-579wv" podUID="dfa4a79b-32ff-457d-bc81-622ee6d5140b" containerName="manager" containerID="cri-o://3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803" gracePeriod=30
Jul 28 20:58:36.947290 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:36.947267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:58:37.012181 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.012151    2578 generic.go:356] "Generic (PLEG): container finished" podID="dfa4a79b-32ff-457d-bc81-622ee6d5140b" containerID="3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803" exitCode=0
Jul 28 20:58:37.012366 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.012199    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76686b4476-579wv"
Jul 28 20:58:37.012366 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.012242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76686b4476-579wv" event={"ID":"dfa4a79b-32ff-457d-bc81-622ee6d5140b","Type":"ContainerDied","Data":"3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803"}
Jul 28 20:58:37.012366 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.012309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76686b4476-579wv" event={"ID":"dfa4a79b-32ff-457d-bc81-622ee6d5140b","Type":"ContainerDied","Data":"f27fd1dd4b287f943e04c7ccc82c5c0060b79ad7ef14b862b2e2824693a4abac"}
Jul 28 20:58:37.012366 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.012341    2578 scope.go:117] "RemoveContainer" containerID="3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803"
Jul 28 20:58:37.020782 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.020764    2578 scope.go:117] "RemoveContainer" containerID="3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803"
Jul 28 20:58:37.021039 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:58:37.021019    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803\": container with ID starting with 3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803 not found: ID does not exist" containerID="3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803"
Jul 28 20:58:37.021127 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.021052    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803"} err="failed to get container status \"3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803\": rpc error: code = NotFound desc = could not find container \"3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803\": container with ID starting with 3649cfc107e94ee6491b09fc430d2249c4319e8d8b2720dcb33afc5513f8d803 not found: ID does not exist"
Jul 28 20:58:37.055357 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.055327    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcsg9\" (UniqueName: \"kubernetes.io/projected/dfa4a79b-32ff-457d-bc81-622ee6d5140b-kube-api-access-bcsg9\") pod \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\" (UID: \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\") "
Jul 28 20:58:37.055494 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.055373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfa4a79b-32ff-457d-bc81-622ee6d5140b-webhook-cert\") pod \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\" (UID: \"dfa4a79b-32ff-457d-bc81-622ee6d5140b\") "
Jul 28 20:58:37.057374 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.057350    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa4a79b-32ff-457d-bc81-622ee6d5140b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dfa4a79b-32ff-457d-bc81-622ee6d5140b" (UID: "dfa4a79b-32ff-457d-bc81-622ee6d5140b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:58:37.057476 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.057415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa4a79b-32ff-457d-bc81-622ee6d5140b-kube-api-access-bcsg9" (OuterVolumeSpecName: "kube-api-access-bcsg9") pod "dfa4a79b-32ff-457d-bc81-622ee6d5140b" (UID: "dfa4a79b-32ff-457d-bc81-622ee6d5140b"). InnerVolumeSpecName "kube-api-access-bcsg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:58:37.156496 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.156462    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcsg9\" (UniqueName: \"kubernetes.io/projected/dfa4a79b-32ff-457d-bc81-622ee6d5140b-kube-api-access-bcsg9\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:58:37.156496 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.156490    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfa4a79b-32ff-457d-bc81-622ee6d5140b-webhook-cert\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:58:37.335024 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.334984    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76686b4476-579wv"]
Jul 28 20:58:37.336680 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.336660    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76686b4476-579wv"]
Jul 28 20:58:37.529506 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:58:37.529464    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfa4a79b-32ff-457d-bc81-622ee6d5140b" path="/var/lib/kubelet/pods/dfa4a79b-32ff-457d-bc81-622ee6d5140b/volumes"
Jul 28 20:59:38.157862 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.157828    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76686b4476-vxkf7"]
Jul 28 20:59:38.158281 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.158230    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa4a79b-32ff-457d-bc81-622ee6d5140b" containerName="manager"
Jul 28 20:59:38.158281 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.158243    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa4a79b-32ff-457d-bc81-622ee6d5140b" containerName="manager"
Jul 28 20:59:38.158357 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.158308    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa4a79b-32ff-457d-bc81-622ee6d5140b" containerName="manager"
Jul 28 20:59:38.161235 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.161215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:38.164593 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.164572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 20:59:38.164593 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.164592    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 20:59:38.164756 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.164572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8q6cc\""
Jul 28 20:59:38.170726 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.170705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76686b4476-vxkf7"]
Jul 28 20:59:38.326059 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.326028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-webhook-cert\") pod \"maas-controller-76686b4476-vxkf7\" (UID: \"b44e0880-0aef-4f3b-85a4-83a480ee0cc0\") " pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:38.326238 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.326100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrh5q\" (UniqueName: \"kubernetes.io/projected/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-kube-api-access-lrh5q\") pod \"maas-controller-76686b4476-vxkf7\" (UID: \"b44e0880-0aef-4f3b-85a4-83a480ee0cc0\") " pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:38.427209 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.427137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrh5q\" (UniqueName: \"kubernetes.io/projected/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-kube-api-access-lrh5q\") pod \"maas-controller-76686b4476-vxkf7\" (UID: \"b44e0880-0aef-4f3b-85a4-83a480ee0cc0\") " pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:38.427335 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.427221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-webhook-cert\") pod \"maas-controller-76686b4476-vxkf7\" (UID: \"b44e0880-0aef-4f3b-85a4-83a480ee0cc0\") " pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:38.429722 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.429703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-webhook-cert\") pod \"maas-controller-76686b4476-vxkf7\" (UID: \"b44e0880-0aef-4f3b-85a4-83a480ee0cc0\") " pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:38.436254 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.436227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrh5q\" (UniqueName: \"kubernetes.io/projected/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-kube-api-access-lrh5q\") pod \"maas-controller-76686b4476-vxkf7\" (UID: \"b44e0880-0aef-4f3b-85a4-83a480ee0cc0\") " pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:38.471964 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.471940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:38.605894 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:38.605869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76686b4476-vxkf7"]
Jul 28 20:59:38.609550 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:59:38.609514    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44e0880_0aef_4f3b_85a4_83a480ee0cc0.slice/crio-ff8fc9a697c69987921ac5b5a409a308e8899292f7124cc3d824c6a02c0fa18d WatchSource:0}: Error finding container ff8fc9a697c69987921ac5b5a409a308e8899292f7124cc3d824c6a02c0fa18d: Status 404 returned error can't find the container with id ff8fc9a697c69987921ac5b5a409a308e8899292f7124cc3d824c6a02c0fa18d
Jul 28 20:59:39.304150 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:39.304116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76686b4476-vxkf7" event={"ID":"b44e0880-0aef-4f3b-85a4-83a480ee0cc0","Type":"ContainerStarted","Data":"ff8fc9a697c69987921ac5b5a409a308e8899292f7124cc3d824c6a02c0fa18d"}
Jul 28 20:59:40.309664 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:40.309631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76686b4476-vxkf7" event={"ID":"b44e0880-0aef-4f3b-85a4-83a480ee0cc0","Type":"ContainerStarted","Data":"63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b"}
Jul 28 20:59:40.310159 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:40.309754    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:40.331317 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:40.331271    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76686b4476-vxkf7" podStartSLOduration=1.544381585 podStartE2EDuration="2.331258081s" totalImagesPullingTime="786.876496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:59:40.32978201 +0000 UTC m=+869.184625920" watchObservedRunningTime="2026-07-28 20:59:40.331258081 +0000 UTC m=+869.186101992"
Jul 28 20:59:51.316595 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:51.316564    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:55.996894 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:55.996853    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76686b4476-vxkf7"]
Jul 28 20:59:55.997379 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:55.997088    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-76686b4476-vxkf7" podUID="b44e0880-0aef-4f3b-85a4-83a480ee0cc0" containerName="manager" containerID="cri-o://63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b" gracePeriod=30
Jul 28 20:59:56.247468 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.247409    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:56.295044 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.295009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrh5q\" (UniqueName: \"kubernetes.io/projected/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-kube-api-access-lrh5q\") pod \"b44e0880-0aef-4f3b-85a4-83a480ee0cc0\" (UID: \"b44e0880-0aef-4f3b-85a4-83a480ee0cc0\") "
Jul 28 20:59:56.295268 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.295239    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-webhook-cert\") pod \"b44e0880-0aef-4f3b-85a4-83a480ee0cc0\" (UID: \"b44e0880-0aef-4f3b-85a4-83a480ee0cc0\") "
Jul 28 20:59:56.297427 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.297394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-kube-api-access-lrh5q" (OuterVolumeSpecName: "kube-api-access-lrh5q") pod "b44e0880-0aef-4f3b-85a4-83a480ee0cc0" (UID: "b44e0880-0aef-4f3b-85a4-83a480ee0cc0"). InnerVolumeSpecName "kube-api-access-lrh5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:59:56.297548 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.297538    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b44e0880-0aef-4f3b-85a4-83a480ee0cc0" (UID: "b44e0880-0aef-4f3b-85a4-83a480ee0cc0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:59:56.385381 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.385341    2578 generic.go:356] "Generic (PLEG): container finished" podID="b44e0880-0aef-4f3b-85a4-83a480ee0cc0" containerID="63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b" exitCode=0
Jul 28 20:59:56.385548 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.385391    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76686b4476-vxkf7"
Jul 28 20:59:56.385548 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.385430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76686b4476-vxkf7" event={"ID":"b44e0880-0aef-4f3b-85a4-83a480ee0cc0","Type":"ContainerDied","Data":"63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b"}
Jul 28 20:59:56.385548 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.385496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76686b4476-vxkf7" event={"ID":"b44e0880-0aef-4f3b-85a4-83a480ee0cc0","Type":"ContainerDied","Data":"ff8fc9a697c69987921ac5b5a409a308e8899292f7124cc3d824c6a02c0fa18d"}
Jul 28 20:59:56.385548 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.385527    2578 scope.go:117] "RemoveContainer" containerID="63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b"
Jul 28 20:59:56.394528 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.394512    2578 scope.go:117] "RemoveContainer" containerID="63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b"
Jul 28 20:59:56.394841 ip-10-0-138-164 kubenswrapper[2578]: E0728 20:59:56.394819    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b\": container with ID starting with 63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b not found: ID does not exist" containerID="63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b"
Jul 28 20:59:56.394891 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.394850    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b"} err="failed to get container status \"63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b\": rpc error: code = NotFound desc = could not find container \"63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b\": container with ID starting with 63a2e7e9a0280438f7b9e585bcf09d1e2952621c068061e498fdd76e7d9be26b not found: ID does not exist"
Jul 28 20:59:56.396310 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.396294    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrh5q\" (UniqueName: \"kubernetes.io/projected/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-kube-api-access-lrh5q\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:59:56.396362 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.396314    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b44e0880-0aef-4f3b-85a4-83a480ee0cc0-webhook-cert\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 20:59:56.406760 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.406712    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76686b4476-vxkf7"]
Jul 28 20:59:56.408745 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:56.408727    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76686b4476-vxkf7"]
Jul 28 20:59:57.207144 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.207111    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76686b4476-w8rl4"]
Jul 28 20:59:57.207544 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.207458    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44e0880-0aef-4f3b-85a4-83a480ee0cc0" containerName="manager"
Jul 28 20:59:57.207544 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.207469    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44e0880-0aef-4f3b-85a4-83a480ee0cc0" containerName="manager"
Jul 28 20:59:57.207544 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.207533    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44e0880-0aef-4f3b-85a4-83a480ee0cc0" containerName="manager"
Jul 28 20:59:57.211677 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.211657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76686b4476-w8rl4"
Jul 28 20:59:57.215562 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.215542    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 20:59:57.215667 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.215567    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 20:59:57.215667 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.215581    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8q6cc\""
Jul 28 20:59:57.220952 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.220929    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76686b4476-w8rl4"]
Jul 28 20:59:57.303746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.303667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9152958-ebe5-4552-b661-45463b8c6e07-webhook-cert\") pod \"maas-controller-76686b4476-w8rl4\" (UID: \"d9152958-ebe5-4552-b661-45463b8c6e07\") " pod="opendatahub/maas-controller-76686b4476-w8rl4"
Jul 28 20:59:57.303746 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.303707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gx2h\" (UniqueName: \"kubernetes.io/projected/d9152958-ebe5-4552-b661-45463b8c6e07-kube-api-access-4gx2h\") pod \"maas-controller-76686b4476-w8rl4\" (UID: \"d9152958-ebe5-4552-b661-45463b8c6e07\") " pod="opendatahub/maas-controller-76686b4476-w8rl4"
Jul 28 20:59:57.404357 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.404323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9152958-ebe5-4552-b661-45463b8c6e07-webhook-cert\") pod \"maas-controller-76686b4476-w8rl4\" (UID: \"d9152958-ebe5-4552-b661-45463b8c6e07\") " pod="opendatahub/maas-controller-76686b4476-w8rl4"
Jul 28 20:59:57.404582 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.404376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gx2h\" (UniqueName: \"kubernetes.io/projected/d9152958-ebe5-4552-b661-45463b8c6e07-kube-api-access-4gx2h\") pod \"maas-controller-76686b4476-w8rl4\" (UID: \"d9152958-ebe5-4552-b661-45463b8c6e07\") " pod="opendatahub/maas-controller-76686b4476-w8rl4"
Jul 28 20:59:57.407570 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.407540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9152958-ebe5-4552-b661-45463b8c6e07-webhook-cert\") pod \"maas-controller-76686b4476-w8rl4\" (UID: \"d9152958-ebe5-4552-b661-45463b8c6e07\") " pod="opendatahub/maas-controller-76686b4476-w8rl4"
Jul 28 20:59:57.413809 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.413784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gx2h\" (UniqueName: \"kubernetes.io/projected/d9152958-ebe5-4552-b661-45463b8c6e07-kube-api-access-4gx2h\") pod \"maas-controller-76686b4476-w8rl4\" (UID: \"d9152958-ebe5-4552-b661-45463b8c6e07\") " pod="opendatahub/maas-controller-76686b4476-w8rl4"
Jul 28 20:59:57.522430 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.522396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76686b4476-w8rl4"
Jul 28 20:59:57.530320 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.530296    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44e0880-0aef-4f3b-85a4-83a480ee0cc0" path="/var/lib/kubelet/pods/b44e0880-0aef-4f3b-85a4-83a480ee0cc0/volumes"
Jul 28 20:59:57.865106 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:57.864994    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76686b4476-w8rl4"]
Jul 28 20:59:57.867719 ip-10-0-138-164 kubenswrapper[2578]: W0728 20:59:57.867695    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9152958_ebe5_4552_b661_45463b8c6e07.slice/crio-61103f9332aa0d9b05fd4df9c7a5fb7f6c19f7d30d1b2340d59209296ecb1c61 WatchSource:0}: Error finding container 61103f9332aa0d9b05fd4df9c7a5fb7f6c19f7d30d1b2340d59209296ecb1c61: Status 404 returned error can't find the container with id 61103f9332aa0d9b05fd4df9c7a5fb7f6c19f7d30d1b2340d59209296ecb1c61
Jul 28 20:59:58.398457 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:58.398421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76686b4476-w8rl4" event={"ID":"d9152958-ebe5-4552-b661-45463b8c6e07","Type":"ContainerStarted","Data":"bfdfc76484253321cd5db38cee73dc18cb5374f5cad58deb18faa87f57c3f8e7"}
Jul 28 20:59:58.398884 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:58.398505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76686b4476-w8rl4" event={"ID":"d9152958-ebe5-4552-b661-45463b8c6e07","Type":"ContainerStarted","Data":"61103f9332aa0d9b05fd4df9c7a5fb7f6c19f7d30d1b2340d59209296ecb1c61"}
Jul 28 20:59:58.398884 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:58.398574    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76686b4476-w8rl4"
Jul 28 20:59:58.416693 ip-10-0-138-164 kubenswrapper[2578]: I0728 20:59:58.416597    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76686b4476-w8rl4" podStartSLOduration=1.132296897 podStartE2EDuration="1.416580954s" totalImagesPullingTime="284.284057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:59:58.414715385 +0000 UTC m=+887.269559296" watchObservedRunningTime="2026-07-28 20:59:58.416580954 +0000 UTC m=+887.271424864"
Jul 28 21:00:09.405308 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:00:09.405228    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76686b4476-w8rl4"
Jul 28 21:04:51.104530 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:51.104421    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"]
Jul 28 21:04:51.107224 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:51.104740    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq" podUID="fc2b74ec-2180-46c5-91d4-c23682a11a07" containerName="manager" containerID="cri-o://e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1" gracePeriod=10
Jul 28 21:04:53.467680 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.467653    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 21:04:53.526213 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.526186    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc2b74ec-2180-46c5-91d4-c23682a11a07-extensions-socket-volume\") pod \"fc2b74ec-2180-46c5-91d4-c23682a11a07\" (UID: \"fc2b74ec-2180-46c5-91d4-c23682a11a07\") "
Jul 28 21:04:53.526406 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.526296    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwkcv\" (UniqueName: \"kubernetes.io/projected/fc2b74ec-2180-46c5-91d4-c23682a11a07-kube-api-access-dwkcv\") pod \"fc2b74ec-2180-46c5-91d4-c23682a11a07\" (UID: \"fc2b74ec-2180-46c5-91d4-c23682a11a07\") "
Jul 28 21:04:53.526533 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.526510    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2b74ec-2180-46c5-91d4-c23682a11a07-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fc2b74ec-2180-46c5-91d4-c23682a11a07" (UID: "fc2b74ec-2180-46c5-91d4-c23682a11a07"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:04:53.528540 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.528506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2b74ec-2180-46c5-91d4-c23682a11a07-kube-api-access-dwkcv" (OuterVolumeSpecName: "kube-api-access-dwkcv") pod "fc2b74ec-2180-46c5-91d4-c23682a11a07" (UID: "fc2b74ec-2180-46c5-91d4-c23682a11a07"). InnerVolumeSpecName "kube-api-access-dwkcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:04:53.627434 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.627395    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc2b74ec-2180-46c5-91d4-c23682a11a07-extensions-socket-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:04:53.627434 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.627426    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwkcv\" (UniqueName: \"kubernetes.io/projected/fc2b74ec-2180-46c5-91d4-c23682a11a07-kube-api-access-dwkcv\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:04:53.783173 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.783140    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc2b74ec-2180-46c5-91d4-c23682a11a07" containerID="e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1" exitCode=0
Jul 28 21:04:53.783352 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.783201    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"
Jul 28 21:04:53.783352 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.783228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq" event={"ID":"fc2b74ec-2180-46c5-91d4-c23682a11a07","Type":"ContainerDied","Data":"e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1"}
Jul 28 21:04:53.783352 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.783318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq" event={"ID":"fc2b74ec-2180-46c5-91d4-c23682a11a07","Type":"ContainerDied","Data":"a81d93749dbe6b883ea2ae85b3b49f02c59b328895ae3860a49f392f84d7073f"}
Jul 28 21:04:53.783352 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.783345    2578 scope.go:117] "RemoveContainer" containerID="e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1"
Jul 28 21:04:53.792714 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.792698    2578 scope.go:117] "RemoveContainer" containerID="e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1"
Jul 28 21:04:53.792957 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:04:53.792941    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1\": container with ID starting with e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1 not found: ID does not exist" containerID="e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1"
Jul 28 21:04:53.793008 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.792964    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1"} err="failed to get container status \"e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1\": rpc error: code = NotFound desc = could not find container \"e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1\": container with ID starting with e712bf4d72b1118e9f4271a490b70ed1d002b91b9ac55e7b1fa379c8f6033ff1 not found: ID does not exist"
Jul 28 21:04:53.800543 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.800521    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"]
Jul 28 21:04:53.804363 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:53.804342    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-94tzq"]
Jul 28 21:04:55.529821 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:04:55.529788    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2b74ec-2180-46c5-91d4-c23682a11a07" path="/var/lib/kubelet/pods/fc2b74ec-2180-46c5-91d4-c23682a11a07/volumes"
Jul 28 21:05:52.350335 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.350295    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"]
Jul 28 21:05:52.350828 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.350653    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2b74ec-2180-46c5-91d4-c23682a11a07" containerName="manager"
Jul 28 21:05:52.350828 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.350663    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2b74ec-2180-46c5-91d4-c23682a11a07" containerName="manager"
Jul 28 21:05:52.350828 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.350747    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2b74ec-2180-46c5-91d4-c23682a11a07" containerName="manager"
Jul 28 21:05:52.353834 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.353818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"
Jul 28 21:05:52.356334 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.356312    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-znxx8\""
Jul 28 21:05:52.366385 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.366367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"]
Jul 28 21:05:52.459370 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.459320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4lq\" (UniqueName: \"kubernetes.io/projected/7a35656c-f8be-4560-b2d8-2db4fe7c845d-kube-api-access-gg4lq\") pod \"kuadrant-operator-controller-manager-86878c769f-pnflf\" (UID: \"7a35656c-f8be-4560-b2d8-2db4fe7c845d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"
Jul 28 21:05:52.459576 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.459492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a35656c-f8be-4560-b2d8-2db4fe7c845d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pnflf\" (UID: \"7a35656c-f8be-4560-b2d8-2db4fe7c845d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"
Jul 28 21:05:52.560944 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.560903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a35656c-f8be-4560-b2d8-2db4fe7c845d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pnflf\" (UID: \"7a35656c-f8be-4560-b2d8-2db4fe7c845d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"
Jul 28 21:05:52.561163 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.560986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4lq\" (UniqueName: \"kubernetes.io/projected/7a35656c-f8be-4560-b2d8-2db4fe7c845d-kube-api-access-gg4lq\") pod \"kuadrant-operator-controller-manager-86878c769f-pnflf\" (UID: \"7a35656c-f8be-4560-b2d8-2db4fe7c845d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"
Jul 28 21:05:52.561303 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.561282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a35656c-f8be-4560-b2d8-2db4fe7c845d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pnflf\" (UID: \"7a35656c-f8be-4560-b2d8-2db4fe7c845d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"
Jul 28 21:05:52.569373 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.569349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4lq\" (UniqueName: \"kubernetes.io/projected/7a35656c-f8be-4560-b2d8-2db4fe7c845d-kube-api-access-gg4lq\") pod \"kuadrant-operator-controller-manager-86878c769f-pnflf\" (UID: \"7a35656c-f8be-4560-b2d8-2db4fe7c845d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"
Jul 28 21:05:52.663579 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.663487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"
Jul 28 21:05:52.812590 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:52.812560    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"]
Jul 28 21:05:52.816971 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:05:52.816946    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a35656c_f8be_4560_b2d8_2db4fe7c845d.slice/crio-9892b3aeee6f7f5ff499520fa4239ca0e4043df648cb91865196bac283003147 WatchSource:0}: Error finding container 9892b3aeee6f7f5ff499520fa4239ca0e4043df648cb91865196bac283003147: Status 404 returned error can't find the container with id 9892b3aeee6f7f5ff499520fa4239ca0e4043df648cb91865196bac283003147
Jul 28 21:05:53.047881 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:53.047844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf" event={"ID":"7a35656c-f8be-4560-b2d8-2db4fe7c845d","Type":"ContainerStarted","Data":"e7e287d2ea14bb589741119cdf4faf1c18f803cb29ac3ed83566c6a6dec59ff4"}
Jul 28 21:05:53.048055 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:53.047902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf" event={"ID":"7a35656c-f8be-4560-b2d8-2db4fe7c845d","Type":"ContainerStarted","Data":"9892b3aeee6f7f5ff499520fa4239ca0e4043df648cb91865196bac283003147"}
Jul 28 21:05:53.048055 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:53.047959    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"
Jul 28 21:05:53.064979 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:05:53.064920    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf" podStartSLOduration=1.06490135 podStartE2EDuration="1.06490135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:05:53.064851687 +0000 UTC m=+1241.919695613" watchObservedRunningTime="2026-07-28 21:05:53.06490135 +0000 UTC m=+1241.919745265"
Jul 28 21:06:04.054663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:06:04.054627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pnflf"
Jul 28 21:13:50.188191 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.188108    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"]
Jul 28 21:13:50.191658 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.191635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:13:50.194261 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.194239    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9fd4a294\""
Jul 28 21:13:50.194395 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.194346    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9fd4a294-dockercfg-m2nhf\""
Jul 28 21:13:50.198823 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.198799    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"]
Jul 28 21:13:50.313659 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.313621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp\" (UID: \"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b\") " pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:13:50.313659 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.313660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdgkj\" (UniqueName: \"kubernetes.io/projected/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-kube-api-access-zdgkj\") pod \"payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp\" (UID: \"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b\") " pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:13:50.414995 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.414959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp\" (UID: \"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b\") " pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:13:50.414995 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.414996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdgkj\" (UniqueName: \"kubernetes.io/projected/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-kube-api-access-zdgkj\") pod \"payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp\" (UID: \"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b\") " pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:13:50.415778 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.415758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp\" (UID: \"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b\") " pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:13:50.423032 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.423002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdgkj\" (UniqueName: \"kubernetes.io/projected/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-kube-api-access-zdgkj\") pod \"payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp\" (UID: \"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b\") " pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:13:50.503011 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.502968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:13:50.641621 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:50.641593    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"]
Jul 28 21:13:50.643400 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:13:50.643373    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c1d3fd_81d5_42f9_bb97_8c78b6e9918b.slice/crio-29b04cbbd604bf3cb9f8b1c78f5fd045703e191a0fef1f013d9e116b4838004a WatchSource:0}: Error finding container 29b04cbbd604bf3cb9f8b1c78f5fd045703e191a0fef1f013d9e116b4838004a: Status 404 returned error can't find the container with id 29b04cbbd604bf3cb9f8b1c78f5fd045703e191a0fef1f013d9e116b4838004a
Jul 28 21:13:51.324382 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:51.324343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp" event={"ID":"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b","Type":"ContainerStarted","Data":"f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e"}
Jul 28 21:13:51.324807 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:51.324411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp" event={"ID":"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b","Type":"ContainerStarted","Data":"29b04cbbd604bf3cb9f8b1c78f5fd045703e191a0fef1f013d9e116b4838004a"}
Jul 28 21:13:51.324807 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:51.324440    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:13:51.341876 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:13:51.341827    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp" podStartSLOduration=1.341811112 podStartE2EDuration="1.341811112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:13:51.340441906 +0000 UTC m=+1720.195285818" watchObservedRunningTime="2026-07-28 21:13:51.341811112 +0000 UTC m=+1720.196655076"
Jul 28 21:14:02.330979 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:02.330945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:14:20.881735 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:20.881702    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"]
Jul 28 21:14:20.882281 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:20.881949    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp" podUID="c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b" containerName="payload-processing" containerID="cri-o://f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e" gracePeriod=30
Jul 28 21:14:21.138106 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.138022    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:14:21.187917 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.187885    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdgkj\" (UniqueName: \"kubernetes.io/projected/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-kube-api-access-zdgkj\") pod \"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b\" (UID: \"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b\") "
Jul 28 21:14:21.188291 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.187951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-plugins-config-volume\") pod \"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b\" (UID: \"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b\") "
Jul 28 21:14:21.188422 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.188364    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b" (UID: "c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:14:21.188642 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.188621    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:14:21.190438 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.190408    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-kube-api-access-zdgkj" (OuterVolumeSpecName: "kube-api-access-zdgkj") pod "c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b" (UID: "c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b"). InnerVolumeSpecName "kube-api-access-zdgkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:14:21.289578 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.289541    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdgkj\" (UniqueName: \"kubernetes.io/projected/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b-kube-api-access-zdgkj\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:14:21.480744 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.480712    2578 generic.go:356] "Generic (PLEG): container finished" podID="c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b" containerID="f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e" exitCode=0
Jul 28 21:14:21.480981 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.480759    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"
Jul 28 21:14:21.480981 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.480785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp" event={"ID":"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b","Type":"ContainerDied","Data":"f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e"}
Jul 28 21:14:21.480981 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.480857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp" event={"ID":"c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b","Type":"ContainerDied","Data":"29b04cbbd604bf3cb9f8b1c78f5fd045703e191a0fef1f013d9e116b4838004a"}
Jul 28 21:14:21.480981 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.480879    2578 scope.go:117] "RemoveContainer" containerID="f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e"
Jul 28 21:14:21.490179 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.490163    2578 scope.go:117] "RemoveContainer" containerID="f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e"
Jul 28 21:14:21.490410 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:14:21.490394    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e\": container with ID starting with f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e not found: ID does not exist" containerID="f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e"
Jul 28 21:14:21.490479 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.490417    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e"} err="failed to get container status \"f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e\": rpc error: code = NotFound desc = could not find container \"f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e\": container with ID starting with f467a5053479d045428582c2d561779a3c8cd6ccd0747eb791974ef4c91c7c5e not found: ID does not exist"
Jul 28 21:14:21.502147 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.502127    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"]
Jul 28 21:14:21.505849 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.505830    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9fd4a294-b5dc4f745-8lxlp"]
Jul 28 21:14:21.529799 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:21.529776    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b" path="/var/lib/kubelet/pods/c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b/volumes"
Jul 28 21:14:25.349849 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.349814    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"]
Jul 28 21:14:25.350242 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.350208    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b" containerName="payload-processing"
Jul 28 21:14:25.350242 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.350219    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b" containerName="payload-processing"
Jul 28 21:14:25.350316 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.350273    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c1d3fd-81d5-42f9-bb97-8c78b6e9918b" containerName="payload-processing"
Jul 28 21:14:25.354608 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.354588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:25.357030 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.357006    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3c452994-dockercfg-bjj5w\""
Jul 28 21:14:25.357198 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.357011    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3c452994\""
Jul 28 21:14:25.360413 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.360390    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"]
Jul 28 21:14:25.431211 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.431177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvghg\" (UniqueName: \"kubernetes.io/projected/7c64fc10-87a5-4a57-b24e-d55a24f28677-kube-api-access-jvghg\") pod \"payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2\" (UID: \"7c64fc10-87a5-4a57-b24e-d55a24f28677\") " pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:25.431371 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.431295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c64fc10-87a5-4a57-b24e-d55a24f28677-plugins-config-volume\") pod \"payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2\" (UID: \"7c64fc10-87a5-4a57-b24e-d55a24f28677\") " pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:25.532353 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.532325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvghg\" (UniqueName: \"kubernetes.io/projected/7c64fc10-87a5-4a57-b24e-d55a24f28677-kube-api-access-jvghg\") pod \"payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2\" (UID: \"7c64fc10-87a5-4a57-b24e-d55a24f28677\") " pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:25.532519 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.532436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c64fc10-87a5-4a57-b24e-d55a24f28677-plugins-config-volume\") pod \"payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2\" (UID: \"7c64fc10-87a5-4a57-b24e-d55a24f28677\") " pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:25.533221 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.533189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c64fc10-87a5-4a57-b24e-d55a24f28677-plugins-config-volume\") pod \"payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2\" (UID: \"7c64fc10-87a5-4a57-b24e-d55a24f28677\") " pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:25.540465 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.540440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvghg\" (UniqueName: \"kubernetes.io/projected/7c64fc10-87a5-4a57-b24e-d55a24f28677-kube-api-access-jvghg\") pod \"payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2\" (UID: \"7c64fc10-87a5-4a57-b24e-d55a24f28677\") " pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:25.666024 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.665941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:25.801310 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:25.801283    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"]
Jul 28 21:14:25.803464 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:14:25.803428    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c64fc10_87a5_4a57_b24e_d55a24f28677.slice/crio-fa34b53239c0561db597408b0732100e95d455326117f0dd2b997929dd0e9af0 WatchSource:0}: Error finding container fa34b53239c0561db597408b0732100e95d455326117f0dd2b997929dd0e9af0: Status 404 returned error can't find the container with id fa34b53239c0561db597408b0732100e95d455326117f0dd2b997929dd0e9af0
Jul 28 21:14:26.508760 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:26.508725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2" event={"ID":"7c64fc10-87a5-4a57-b24e-d55a24f28677","Type":"ContainerStarted","Data":"3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3"}
Jul 28 21:14:26.509166 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:26.508782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2" event={"ID":"7c64fc10-87a5-4a57-b24e-d55a24f28677","Type":"ContainerStarted","Data":"fa34b53239c0561db597408b0732100e95d455326117f0dd2b997929dd0e9af0"}
Jul 28 21:14:26.509166 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:26.508823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:26.524056 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:26.524009    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2" podStartSLOduration=1.523995478 podStartE2EDuration="1.523995478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:14:26.523510095 +0000 UTC m=+1755.378354019" watchObservedRunningTime="2026-07-28 21:14:26.523995478 +0000 UTC m=+1755.378839388"
Jul 28 21:14:37.515928 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:37.515899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:55.209830 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.209798    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"]
Jul 28 21:14:55.212618 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.210101    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2" podUID="7c64fc10-87a5-4a57-b24e-d55a24f28677" containerName="payload-processing" containerID="cri-o://3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3" gracePeriod=30
Jul 28 21:14:55.469289 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.469220    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:55.600711 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.600680    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c64fc10-87a5-4a57-b24e-d55a24f28677-plugins-config-volume\") pod \"7c64fc10-87a5-4a57-b24e-d55a24f28677\" (UID: \"7c64fc10-87a5-4a57-b24e-d55a24f28677\") "
Jul 28 21:14:55.600934 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.600748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvghg\" (UniqueName: \"kubernetes.io/projected/7c64fc10-87a5-4a57-b24e-d55a24f28677-kube-api-access-jvghg\") pod \"7c64fc10-87a5-4a57-b24e-d55a24f28677\" (UID: \"7c64fc10-87a5-4a57-b24e-d55a24f28677\") "
Jul 28 21:14:55.601160 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.601129    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c64fc10-87a5-4a57-b24e-d55a24f28677-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c64fc10-87a5-4a57-b24e-d55a24f28677" (UID: "7c64fc10-87a5-4a57-b24e-d55a24f28677"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:14:55.603015 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.602987    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c64fc10-87a5-4a57-b24e-d55a24f28677-kube-api-access-jvghg" (OuterVolumeSpecName: "kube-api-access-jvghg") pod "7c64fc10-87a5-4a57-b24e-d55a24f28677" (UID: "7c64fc10-87a5-4a57-b24e-d55a24f28677"). InnerVolumeSpecName "kube-api-access-jvghg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:14:55.643592 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.643554    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c64fc10-87a5-4a57-b24e-d55a24f28677" containerID="3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3" exitCode=0
Jul 28 21:14:55.643740 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.643605    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"
Jul 28 21:14:55.643740 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.643658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2" event={"ID":"7c64fc10-87a5-4a57-b24e-d55a24f28677","Type":"ContainerDied","Data":"3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3"}
Jul 28 21:14:55.643740 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.643701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2" event={"ID":"7c64fc10-87a5-4a57-b24e-d55a24f28677","Type":"ContainerDied","Data":"fa34b53239c0561db597408b0732100e95d455326117f0dd2b997929dd0e9af0"}
Jul 28 21:14:55.643740 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.643722    2578 scope.go:117] "RemoveContainer" containerID="3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3"
Jul 28 21:14:55.652753 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.652736    2578 scope.go:117] "RemoveContainer" containerID="3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3"
Jul 28 21:14:55.653034 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:14:55.653019    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3\": container with ID starting with 3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3 not found: ID does not exist" containerID="3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3"
Jul 28 21:14:55.653098 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.653043    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3"} err="failed to get container status \"3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3\": rpc error: code = NotFound desc = could not find container \"3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3\": container with ID starting with 3c99ba6ddf9e0c83ffb40b8cbd10dc2e22a699d9a9bd4f99da8596c34c22f5d3 not found: ID does not exist"
Jul 28 21:14:55.666225 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.666195    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"]
Jul 28 21:14:55.671743 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.671718    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c452994-846b4b5798-9tdb2"]
Jul 28 21:14:55.701888 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.701870    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c64fc10-87a5-4a57-b24e-d55a24f28677-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:14:55.701888 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:55.701890    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvghg\" (UniqueName: \"kubernetes.io/projected/7c64fc10-87a5-4a57-b24e-d55a24f28677-kube-api-access-jvghg\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:14:57.529157 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:14:57.529126    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c64fc10-87a5-4a57-b24e-d55a24f28677" path="/var/lib/kubelet/pods/7c64fc10-87a5-4a57-b24e-d55a24f28677/volumes"
Jul 28 21:15:00.511069 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.511028    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"]
Jul 28 21:15:00.511680 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.511657    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c64fc10-87a5-4a57-b24e-d55a24f28677" containerName="payload-processing"
Jul 28 21:15:00.511736 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.511686    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c64fc10-87a5-4a57-b24e-d55a24f28677" containerName="payload-processing"
Jul 28 21:15:00.511821 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.511809    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c64fc10-87a5-4a57-b24e-d55a24f28677" containerName="payload-processing"
Jul 28 21:15:00.516877 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.516850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:00.519595 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.519568    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-42733177\""
Jul 28 21:15:00.519727 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.519643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-42733177-dockercfg-f9ftx\""
Jul 28 21:15:00.525816 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.525793    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"]
Jul 28 21:15:00.540258 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.540220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5242e89b-1a5f-4619-98d9-26f27a57e3f6-plugins-config-volume\") pod \"payload-processing-e2e-derive-42733177-75f4c5894-pcsmn\" (UID: \"5242e89b-1a5f-4619-98d9-26f27a57e3f6\") " pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:00.540258 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.540263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82th\" (UniqueName: \"kubernetes.io/projected/5242e89b-1a5f-4619-98d9-26f27a57e3f6-kube-api-access-c82th\") pod \"payload-processing-e2e-derive-42733177-75f4c5894-pcsmn\" (UID: \"5242e89b-1a5f-4619-98d9-26f27a57e3f6\") " pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:00.640924 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.640880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5242e89b-1a5f-4619-98d9-26f27a57e3f6-plugins-config-volume\") pod \"payload-processing-e2e-derive-42733177-75f4c5894-pcsmn\" (UID: \"5242e89b-1a5f-4619-98d9-26f27a57e3f6\") " pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:00.641156 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.640935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82th\" (UniqueName: \"kubernetes.io/projected/5242e89b-1a5f-4619-98d9-26f27a57e3f6-kube-api-access-c82th\") pod \"payload-processing-e2e-derive-42733177-75f4c5894-pcsmn\" (UID: \"5242e89b-1a5f-4619-98d9-26f27a57e3f6\") " pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:00.641874 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.641847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5242e89b-1a5f-4619-98d9-26f27a57e3f6-plugins-config-volume\") pod \"payload-processing-e2e-derive-42733177-75f4c5894-pcsmn\" (UID: \"5242e89b-1a5f-4619-98d9-26f27a57e3f6\") " pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:00.649994 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.649968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82th\" (UniqueName: \"kubernetes.io/projected/5242e89b-1a5f-4619-98d9-26f27a57e3f6-kube-api-access-c82th\") pod \"payload-processing-e2e-derive-42733177-75f4c5894-pcsmn\" (UID: \"5242e89b-1a5f-4619-98d9-26f27a57e3f6\") " pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:00.829613 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.829515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:00.973526 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:00.973497    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"]
Jul 28 21:15:00.975567 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:15:00.975542    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5242e89b_1a5f_4619_98d9_26f27a57e3f6.slice/crio-50ad42e188ac8c0110dcfbe66cecc0f773e51b2c84304b4a00eb53e4024dd4a0 WatchSource:0}: Error finding container 50ad42e188ac8c0110dcfbe66cecc0f773e51b2c84304b4a00eb53e4024dd4a0: Status 404 returned error can't find the container with id 50ad42e188ac8c0110dcfbe66cecc0f773e51b2c84304b4a00eb53e4024dd4a0
Jul 28 21:15:01.673773 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:01.673736    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn" event={"ID":"5242e89b-1a5f-4619-98d9-26f27a57e3f6","Type":"ContainerStarted","Data":"ac66019d8b3c643d05743f2c4df1e2c847722f393e384d57a31a70b8ac4ad621"}
Jul 28 21:15:01.674187 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:01.673790    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn" event={"ID":"5242e89b-1a5f-4619-98d9-26f27a57e3f6","Type":"ContainerStarted","Data":"50ad42e188ac8c0110dcfbe66cecc0f773e51b2c84304b4a00eb53e4024dd4a0"}
Jul 28 21:15:01.674187 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:01.673874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:01.696641 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:01.696581    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn" podStartSLOduration=1.696566148 podStartE2EDuration="1.696566148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:15:01.69366383 +0000 UTC m=+1790.548507742" watchObservedRunningTime="2026-07-28 21:15:01.696566148 +0000 UTC m=+1790.551410059"
Jul 28 21:15:12.680537 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:12.680508    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:30.652745 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:30.652708    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"]
Jul 28 21:15:30.653331 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:30.653006    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn" podUID="5242e89b-1a5f-4619-98d9-26f27a57e3f6" containerName="payload-processing" containerID="cri-o://ac66019d8b3c643d05743f2c4df1e2c847722f393e384d57a31a70b8ac4ad621" gracePeriod=30
Jul 28 21:15:30.816875 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:30.816840    2578 generic.go:356] "Generic (PLEG): container finished" podID="5242e89b-1a5f-4619-98d9-26f27a57e3f6" containerID="ac66019d8b3c643d05743f2c4df1e2c847722f393e384d57a31a70b8ac4ad621" exitCode=0
Jul 28 21:15:30.817057 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:30.816929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn" event={"ID":"5242e89b-1a5f-4619-98d9-26f27a57e3f6","Type":"ContainerDied","Data":"ac66019d8b3c643d05743f2c4df1e2c847722f393e384d57a31a70b8ac4ad621"}
Jul 28 21:15:30.915785 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:30.915722    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:31.017190 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.017157    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5242e89b-1a5f-4619-98d9-26f27a57e3f6-plugins-config-volume\") pod \"5242e89b-1a5f-4619-98d9-26f27a57e3f6\" (UID: \"5242e89b-1a5f-4619-98d9-26f27a57e3f6\") "
Jul 28 21:15:31.017359 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.017214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c82th\" (UniqueName: \"kubernetes.io/projected/5242e89b-1a5f-4619-98d9-26f27a57e3f6-kube-api-access-c82th\") pod \"5242e89b-1a5f-4619-98d9-26f27a57e3f6\" (UID: \"5242e89b-1a5f-4619-98d9-26f27a57e3f6\") "
Jul 28 21:15:31.017519 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.017491    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5242e89b-1a5f-4619-98d9-26f27a57e3f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5242e89b-1a5f-4619-98d9-26f27a57e3f6" (UID: "5242e89b-1a5f-4619-98d9-26f27a57e3f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:15:31.019640 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.019614    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5242e89b-1a5f-4619-98d9-26f27a57e3f6-kube-api-access-c82th" (OuterVolumeSpecName: "kube-api-access-c82th") pod "5242e89b-1a5f-4619-98d9-26f27a57e3f6" (UID: "5242e89b-1a5f-4619-98d9-26f27a57e3f6"). InnerVolumeSpecName "kube-api-access-c82th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:15:31.118104 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.118043    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5242e89b-1a5f-4619-98d9-26f27a57e3f6-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:15:31.118104 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.118071    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c82th\" (UniqueName: \"kubernetes.io/projected/5242e89b-1a5f-4619-98d9-26f27a57e3f6-kube-api-access-c82th\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:15:31.822275 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.822184    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn" event={"ID":"5242e89b-1a5f-4619-98d9-26f27a57e3f6","Type":"ContainerDied","Data":"50ad42e188ac8c0110dcfbe66cecc0f773e51b2c84304b4a00eb53e4024dd4a0"}
Jul 28 21:15:31.822275 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.822200    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"
Jul 28 21:15:31.822275 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.822244    2578 scope.go:117] "RemoveContainer" containerID="ac66019d8b3c643d05743f2c4df1e2c847722f393e384d57a31a70b8ac4ad621"
Jul 28 21:15:31.839663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.839639    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"]
Jul 28 21:15:31.841424 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:31.841404    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42733177-75f4c5894-pcsmn"]
Jul 28 21:15:33.530623 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:15:33.530592    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5242e89b-1a5f-4619-98d9-26f27a57e3f6" path="/var/lib/kubelet/pods/5242e89b-1a5f-4619-98d9-26f27a57e3f6/volumes"
Jul 28 21:19:14.635966 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.635928    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"]
Jul 28 21:19:14.636456 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.636321    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5242e89b-1a5f-4619-98d9-26f27a57e3f6" containerName="payload-processing"
Jul 28 21:19:14.636456 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.636335    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5242e89b-1a5f-4619-98d9-26f27a57e3f6" containerName="payload-processing"
Jul 28 21:19:14.636456 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.636402    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5242e89b-1a5f-4619-98d9-26f27a57e3f6" containerName="payload-processing"
Jul 28 21:19:14.639771 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.639750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.643229 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.643206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2dd70051\"/\"default-dockercfg-9zpb9\""
Jul 28 21:19:14.643375 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.643206    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2dd70051\"/\"openshift-service-ca.crt\""
Jul 28 21:19:14.643375 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.643211    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2dd70051\"/\"e2e-lifecycle-model-2dd70051-kserve-self-signed-certs\""
Jul 28 21:19:14.643375 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.643216    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2dd70051\"/\"kube-root-ca.crt\""
Jul 28 21:19:14.652550 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.650797    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"]
Jul 28 21:19:14.761739 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.761693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2825\" (UniqueName: \"kubernetes.io/projected/0f71672e-ebe0-4ed9-addd-06d12cc03864-kube-api-access-f2825\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.761915 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.761750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-tmp-dir\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.761915 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.761820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-model-cache\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.762014 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.761913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-home\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.762014 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.761977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f71672e-ebe0-4ed9-addd-06d12cc03864-tls-certs\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.762141 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.762021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-dshm\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.863037 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.863006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-home\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.863232 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.863054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f71672e-ebe0-4ed9-addd-06d12cc03864-tls-certs\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.863232 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.863132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-dshm\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.863232 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.863191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2825\" (UniqueName: \"kubernetes.io/projected/0f71672e-ebe0-4ed9-addd-06d12cc03864-kube-api-access-f2825\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.863232 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.863225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-tmp-dir\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.863403 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.863256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-model-cache\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.863494 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.863471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-home\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.863609 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.863584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-tmp-dir\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.863609 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.863605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-model-cache\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.865549 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.865527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-dshm\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.865858 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.865839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f71672e-ebe0-4ed9-addd-06d12cc03864-tls-certs\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.871916 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.871893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2825\" (UniqueName: \"kubernetes.io/projected/0f71672e-ebe0-4ed9-addd-06d12cc03864-kube-api-access-f2825\") pod \"e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") " pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:14.949293 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:14.949210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:15.082525 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:15.082499    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"]
Jul 28 21:19:15.084509 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:19:15.084474    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f71672e_ebe0_4ed9_addd_06d12cc03864.slice/crio-5edff3f089169987c82a059a4083af0ada396546cdfa5ab2bc73e3bc903512aa WatchSource:0}: Error finding container 5edff3f089169987c82a059a4083af0ada396546cdfa5ab2bc73e3bc903512aa: Status 404 returned error can't find the container with id 5edff3f089169987c82a059a4083af0ada396546cdfa5ab2bc73e3bc903512aa
Jul 28 21:19:15.876776 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:15.876740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq" event={"ID":"0f71672e-ebe0-4ed9-addd-06d12cc03864","Type":"ContainerStarted","Data":"21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e"}
Jul 28 21:19:15.877267 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:15.876796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq" event={"ID":"0f71672e-ebe0-4ed9-addd-06d12cc03864","Type":"ContainerStarted","Data":"5edff3f089169987c82a059a4083af0ada396546cdfa5ab2bc73e3bc903512aa"}
Jul 28 21:19:15.897219 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:15.897173    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq" podStartSLOduration=1.897156754 podStartE2EDuration="1.897156754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:19:15.895320685 +0000 UTC m=+2044.750164597" watchObservedRunningTime="2026-07-28 21:19:15.897156754 +0000 UTC m=+2044.752000679"
Jul 28 21:19:24.950236 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:24.950204    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:24.950721 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:24.950279    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:24.962908 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:24.962879    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:19:25.936645 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:19:25.936611    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:20:07.275886 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:07.275849    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"]
Jul 28 21:20:22.295626 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.295556    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq" podUID="0f71672e-ebe0-4ed9-addd-06d12cc03864" containerName="main" containerID="cri-o://21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e" gracePeriod=45
Jul 28 21:20:22.549879 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.549822    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:20:22.576962 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.576931    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-dshm\") pod \"0f71672e-ebe0-4ed9-addd-06d12cc03864\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") "
Jul 28 21:20:22.577133 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.577001    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-home\") pod \"0f71672e-ebe0-4ed9-addd-06d12cc03864\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") "
Jul 28 21:20:22.577133 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.577036    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f71672e-ebe0-4ed9-addd-06d12cc03864-tls-certs\") pod \"0f71672e-ebe0-4ed9-addd-06d12cc03864\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") "
Jul 28 21:20:22.577133 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.577115    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2825\" (UniqueName: \"kubernetes.io/projected/0f71672e-ebe0-4ed9-addd-06d12cc03864-kube-api-access-f2825\") pod \"0f71672e-ebe0-4ed9-addd-06d12cc03864\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") "
Jul 28 21:20:22.577306 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.577179    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-model-cache\") pod \"0f71672e-ebe0-4ed9-addd-06d12cc03864\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") "
Jul 28 21:20:22.577377 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.577338    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-tmp-dir\") pod \"0f71672e-ebe0-4ed9-addd-06d12cc03864\" (UID: \"0f71672e-ebe0-4ed9-addd-06d12cc03864\") "
Jul 28 21:20:22.577450 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.577426    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f71672e-ebe0-4ed9-addd-06d12cc03864" (UID: "0f71672e-ebe0-4ed9-addd-06d12cc03864"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:20:22.577826 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.577684    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f71672e-ebe0-4ed9-addd-06d12cc03864" (UID: "0f71672e-ebe0-4ed9-addd-06d12cc03864"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:20:22.577826 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.577703    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-model-cache\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:20:22.577826 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.577784    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-home" (OuterVolumeSpecName: "home") pod "0f71672e-ebe0-4ed9-addd-06d12cc03864" (UID: "0f71672e-ebe0-4ed9-addd-06d12cc03864"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:20:22.579383 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.579359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-dshm" (OuterVolumeSpecName: "dshm") pod "0f71672e-ebe0-4ed9-addd-06d12cc03864" (UID: "0f71672e-ebe0-4ed9-addd-06d12cc03864"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:20:22.579825 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.579805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f71672e-ebe0-4ed9-addd-06d12cc03864-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f71672e-ebe0-4ed9-addd-06d12cc03864" (UID: "0f71672e-ebe0-4ed9-addd-06d12cc03864"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:20:22.579879 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.579833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f71672e-ebe0-4ed9-addd-06d12cc03864-kube-api-access-f2825" (OuterVolumeSpecName: "kube-api-access-f2825") pod "0f71672e-ebe0-4ed9-addd-06d12cc03864" (UID: "0f71672e-ebe0-4ed9-addd-06d12cc03864"). InnerVolumeSpecName "kube-api-access-f2825". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:20:22.678840 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.678807    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-tmp-dir\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:20:22.678840 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.678835    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-dshm\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:20:22.678840 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.678843    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f71672e-ebe0-4ed9-addd-06d12cc03864-home\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:20:22.679104 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.678859    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f71672e-ebe0-4ed9-addd-06d12cc03864-tls-certs\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:20:22.679104 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:22.678868    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2825\" (UniqueName: \"kubernetes.io/projected/0f71672e-ebe0-4ed9-addd-06d12cc03864-kube-api-access-f2825\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:20:23.200172 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:23.200135    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f71672e-ebe0-4ed9-addd-06d12cc03864" containerID="21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e" exitCode=0
Jul 28 21:20:23.200346 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:23.200184    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"
Jul 28 21:20:23.200346 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:23.200198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq" event={"ID":"0f71672e-ebe0-4ed9-addd-06d12cc03864","Type":"ContainerDied","Data":"21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e"}
Jul 28 21:20:23.200346 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:23.200237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq" event={"ID":"0f71672e-ebe0-4ed9-addd-06d12cc03864","Type":"ContainerDied","Data":"5edff3f089169987c82a059a4083af0ada396546cdfa5ab2bc73e3bc903512aa"}
Jul 28 21:20:23.200346 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:23.200263    2578 scope.go:117] "RemoveContainer" containerID="21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e"
Jul 28 21:20:23.209486 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:23.209467    2578 scope.go:117] "RemoveContainer" containerID="21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e"
Jul 28 21:20:23.209739 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:20:23.209722    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e\": container with ID starting with 21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e not found: ID does not exist" containerID="21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e"
Jul 28 21:20:23.209781 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:23.209747    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e"} err="failed to get container status \"21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e\": rpc error: code = NotFound desc = could not find container \"21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e\": container with ID starting with 21c1537495d7d74ac9b2fa1e061825604bcdca1fe7e3797042a6095bc9be858e not found: ID does not exist"
Jul 28 21:20:23.221050 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:23.221026    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"]
Jul 28 21:20:23.225120 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:23.225098    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2dd70051/e2e-lifecycle-model-2dd70051-kserve-58b5bf859f-fvhtq"]
Jul 28 21:20:23.529784 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:20:23.529745    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f71672e-ebe0-4ed9-addd-06d12cc03864" path="/var/lib/kubelet/pods/0f71672e-ebe0-4ed9-addd-06d12cc03864/volumes"
Jul 28 21:22:21.582031 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.581995    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"]
Jul 28 21:22:21.582529 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.582361    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f71672e-ebe0-4ed9-addd-06d12cc03864" containerName="main"
Jul 28 21:22:21.582529 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.582373    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f71672e-ebe0-4ed9-addd-06d12cc03864" containerName="main"
Jul 28 21:22:21.582529 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.582443    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f71672e-ebe0-4ed9-addd-06d12cc03864" containerName="main"
Jul 28 21:22:21.585396 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.585378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:22:21.587872 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.587849    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9bf2ee\""
Jul 28 21:22:21.587985 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.587852    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9bf2ee-dockercfg-mcv25\""
Jul 28 21:22:21.594705 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.594681    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"]
Jul 28 21:22:21.598848 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.598829    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"]
Jul 28 21:22:21.602111 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.602096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:22:21.610685 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.610658    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"]
Jul 28 21:22:21.757818 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.757783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk48j\" (UniqueName: \"kubernetes.io/projected/bf6f9721-3b48-428a-898a-1610fc742318-kube-api-access-zk48j\") pod \"payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw\" (UID: \"bf6f9721-3b48-428a-898a-1610fc742318\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:22:21.758015 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.757833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq2h4\" (UniqueName: \"kubernetes.io/projected/6bc355e9-8726-400c-827c-7b9d54d70895-kube-api-access-dq2h4\") pod \"payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w\" (UID: \"6bc355e9-8726-400c-827c-7b9d54d70895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:22:21.758015 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.757916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc355e9-8726-400c-827c-7b9d54d70895-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w\" (UID: \"6bc355e9-8726-400c-827c-7b9d54d70895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:22:21.758015 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.757995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6f9721-3b48-428a-898a-1610fc742318-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw\" (UID: \"bf6f9721-3b48-428a-898a-1610fc742318\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:22:21.859031 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.858930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc355e9-8726-400c-827c-7b9d54d70895-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w\" (UID: \"6bc355e9-8726-400c-827c-7b9d54d70895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:22:21.859031 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.859003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6f9721-3b48-428a-898a-1610fc742318-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw\" (UID: \"bf6f9721-3b48-428a-898a-1610fc742318\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:22:21.859309 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.859057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk48j\" (UniqueName: \"kubernetes.io/projected/bf6f9721-3b48-428a-898a-1610fc742318-kube-api-access-zk48j\") pod \"payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw\" (UID: \"bf6f9721-3b48-428a-898a-1610fc742318\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:22:21.859309 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.859138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq2h4\" (UniqueName: \"kubernetes.io/projected/6bc355e9-8726-400c-827c-7b9d54d70895-kube-api-access-dq2h4\") pod \"payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w\" (UID: \"6bc355e9-8726-400c-827c-7b9d54d70895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:22:21.859730 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.859695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc355e9-8726-400c-827c-7b9d54d70895-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w\" (UID: \"6bc355e9-8726-400c-827c-7b9d54d70895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:22:21.859843 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.859748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6f9721-3b48-428a-898a-1610fc742318-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw\" (UID: \"bf6f9721-3b48-428a-898a-1610fc742318\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:22:21.868046 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.868015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq2h4\" (UniqueName: \"kubernetes.io/projected/6bc355e9-8726-400c-827c-7b9d54d70895-kube-api-access-dq2h4\") pod \"payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w\" (UID: \"6bc355e9-8726-400c-827c-7b9d54d70895\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:22:21.868231 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.868210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk48j\" (UniqueName: \"kubernetes.io/projected/bf6f9721-3b48-428a-898a-1610fc742318-kube-api-access-zk48j\") pod \"payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw\" (UID: \"bf6f9721-3b48-428a-898a-1610fc742318\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:22:21.897224 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.897190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:22:21.913368 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:21.913341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:22:22.083151 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:22.083123    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"]
Jul 28 21:22:22.084876 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:22:22.084839    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6f9721_3b48_428a_898a_1610fc742318.slice/crio-f01051b9af63a83b22546def6e648c743110fb3e0bd8b526ac8df69fafcc7fce WatchSource:0}: Error finding container f01051b9af63a83b22546def6e648c743110fb3e0bd8b526ac8df69fafcc7fce: Status 404 returned error can't find the container with id f01051b9af63a83b22546def6e648c743110fb3e0bd8b526ac8df69fafcc7fce
Jul 28 21:22:22.260869 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:22.260837    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"]
Jul 28 21:22:22.262736 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:22:22.262709    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc355e9_8726_400c_827c_7b9d54d70895.slice/crio-af767e9535239025c8dfa269bc93790836061d5d1bc96a957caa307680409f41 WatchSource:0}: Error finding container af767e9535239025c8dfa269bc93790836061d5d1bc96a957caa307680409f41: Status 404 returned error can't find the container with id af767e9535239025c8dfa269bc93790836061d5d1bc96a957caa307680409f41
Jul 28 21:22:22.774517 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:22.774480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w" event={"ID":"6bc355e9-8726-400c-827c-7b9d54d70895","Type":"ContainerStarted","Data":"aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31"}
Jul 28 21:22:22.774941 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:22.774537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w" event={"ID":"6bc355e9-8726-400c-827c-7b9d54d70895","Type":"ContainerStarted","Data":"af767e9535239025c8dfa269bc93790836061d5d1bc96a957caa307680409f41"}
Jul 28 21:22:22.774941 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:22.774585    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:22:22.776217 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:22.776193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw" event={"ID":"bf6f9721-3b48-428a-898a-1610fc742318","Type":"ContainerStarted","Data":"0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac"}
Jul 28 21:22:22.776330 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:22.776239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw" event={"ID":"bf6f9721-3b48-428a-898a-1610fc742318","Type":"ContainerStarted","Data":"f01051b9af63a83b22546def6e648c743110fb3e0bd8b526ac8df69fafcc7fce"}
Jul 28 21:22:22.776330 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:22.776259    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:22:22.791562 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:22.791508    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w" podStartSLOduration=1.791491718 podStartE2EDuration="1.791491718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:22:22.791199434 +0000 UTC m=+2231.646043345" watchObservedRunningTime="2026-07-28 21:22:22.791491718 +0000 UTC m=+2231.646335630"
Jul 28 21:22:22.808476 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:22.808426    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw" podStartSLOduration=1.808411359 podStartE2EDuration="1.808411359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:22:22.80649697 +0000 UTC m=+2231.661340883" watchObservedRunningTime="2026-07-28 21:22:22.808411359 +0000 UTC m=+2231.663255270"
Jul 28 21:22:33.782390 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:33.782176    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w" podUID="6bc355e9-8726-400c-827c-7b9d54d70895" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.48:9004: connect: connection refused"
Jul 28 21:22:33.782829 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:33.782657    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:22:43.781176 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:22:43.781141    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:23:17.348418 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.348385    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"]
Jul 28 21:23:17.351946 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.351927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.356294 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.356271    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9bf2ee\"/\"test-model-9bf2ee-kserve-self-signed-certs\""
Jul 28 21:23:17.356453 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.356432    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9bf2ee\"/\"kube-root-ca.crt\""
Jul 28 21:23:17.356522 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.356478    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9bf2ee\"/\"openshift-service-ca.crt\""
Jul 28 21:23:17.356522 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.356500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr9vs\" (UniqueName: \"kubernetes.io/projected/2469fa44-5bee-4117-88f7-658a8ee48d6a-kube-api-access-zr9vs\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.356617 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.356479    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9bf2ee\"/\"default-dockercfg-cw7s8\""
Jul 28 21:23:17.356617 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.356580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-tmp-dir\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.356617 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.356607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-home\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.356736 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.356630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-dshm\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.356736 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.356669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-model-cache\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.356736 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.356693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2469fa44-5bee-4117-88f7-658a8ee48d6a-tls-certs\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.364060 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.364040    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"]
Jul 28 21:23:17.457744 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.457705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr9vs\" (UniqueName: \"kubernetes.io/projected/2469fa44-5bee-4117-88f7-658a8ee48d6a-kube-api-access-zr9vs\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.457916 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.457801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-tmp-dir\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.457916 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.457827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-home\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.457916 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.457843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-dshm\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.457916 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.457860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-model-cache\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.457916 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.457876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2469fa44-5bee-4117-88f7-658a8ee48d6a-tls-certs\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.458302 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.458271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-home\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.458440 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.458342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-model-cache\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.458440 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.458372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-tmp-dir\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.460439 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.460415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-dshm\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.460626 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.460607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2469fa44-5bee-4117-88f7-658a8ee48d6a-tls-certs\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.471189 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.471157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr9vs\" (UniqueName: \"kubernetes.io/projected/2469fa44-5bee-4117-88f7-658a8ee48d6a-kube-api-access-zr9vs\") pod \"test-model-9bf2ee-kserve-76c6b4cdc5-khjch\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") " pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.660698 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.660596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:17.800868 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:17.800834    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"]
Jul 28 21:23:17.803736 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:23:17.803713    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2469fa44_5bee_4117_88f7_658a8ee48d6a.slice/crio-5067abe6c68698c496305fede8c38d60c2e6ac883ecba6946cb17a7db0950563 WatchSource:0}: Error finding container 5067abe6c68698c496305fede8c38d60c2e6ac883ecba6946cb17a7db0950563: Status 404 returned error can't find the container with id 5067abe6c68698c496305fede8c38d60c2e6ac883ecba6946cb17a7db0950563
Jul 28 21:23:18.054104 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:18.054044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch" event={"ID":"2469fa44-5bee-4117-88f7-658a8ee48d6a","Type":"ContainerStarted","Data":"2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd"}
Jul 28 21:23:18.054291 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:18.054113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch" event={"ID":"2469fa44-5bee-4117-88f7-658a8ee48d6a","Type":"ContainerStarted","Data":"5067abe6c68698c496305fede8c38d60c2e6ac883ecba6946cb17a7db0950563"}
Jul 28 21:23:18.077752 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:18.077707    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch" podStartSLOduration=1.077692709 podStartE2EDuration="1.077692709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:23:18.07615173 +0000 UTC m=+2286.930995641" watchObservedRunningTime="2026-07-28 21:23:18.077692709 +0000 UTC m=+2286.932536662"
Jul 28 21:23:27.661416 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:27.661372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:27.661884 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:27.661454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:27.673917 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:27.673888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:23:28.112670 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:23:28.112635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:24:06.714558 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:06.714473    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"]
Jul 28 21:24:21.734188 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:21.734145    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch" podUID="2469fa44-5bee-4117-88f7-658a8ee48d6a" containerName="main" containerID="cri-o://2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd" gracePeriod=45
Jul 28 21:24:21.998805 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:21.998745    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:24:22.060922 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.060888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-model-cache\") pod \"2469fa44-5bee-4117-88f7-658a8ee48d6a\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") "
Jul 28 21:24:22.061114 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.060932    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2469fa44-5bee-4117-88f7-658a8ee48d6a-tls-certs\") pod \"2469fa44-5bee-4117-88f7-658a8ee48d6a\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") "
Jul 28 21:24:22.061114 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.060975    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-home\") pod \"2469fa44-5bee-4117-88f7-658a8ee48d6a\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") "
Jul 28 21:24:22.061114 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.060997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr9vs\" (UniqueName: \"kubernetes.io/projected/2469fa44-5bee-4117-88f7-658a8ee48d6a-kube-api-access-zr9vs\") pod \"2469fa44-5bee-4117-88f7-658a8ee48d6a\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") "
Jul 28 21:24:22.061114 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.061030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-dshm\") pod \"2469fa44-5bee-4117-88f7-658a8ee48d6a\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") "
Jul 28 21:24:22.061356 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.061136    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-tmp-dir\") pod \"2469fa44-5bee-4117-88f7-658a8ee48d6a\" (UID: \"2469fa44-5bee-4117-88f7-658a8ee48d6a\") "
Jul 28 21:24:22.061356 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.061283    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "2469fa44-5bee-4117-88f7-658a8ee48d6a" (UID: "2469fa44-5bee-4117-88f7-658a8ee48d6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:24:22.061447 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.061382    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-home" (OuterVolumeSpecName: "home") pod "2469fa44-5bee-4117-88f7-658a8ee48d6a" (UID: "2469fa44-5bee-4117-88f7-658a8ee48d6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:24:22.061447 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.061423    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2469fa44-5bee-4117-88f7-658a8ee48d6a" (UID: "2469fa44-5bee-4117-88f7-658a8ee48d6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:24:22.061605 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.061583    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-model-cache\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:24:22.061605 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.061605    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-home\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:24:22.061769 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.061618    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-tmp-dir\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:24:22.063474 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.063454    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2469fa44-5bee-4117-88f7-658a8ee48d6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2469fa44-5bee-4117-88f7-658a8ee48d6a" (UID: "2469fa44-5bee-4117-88f7-658a8ee48d6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:24:22.063557 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.063514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2469fa44-5bee-4117-88f7-658a8ee48d6a-kube-api-access-zr9vs" (OuterVolumeSpecName: "kube-api-access-zr9vs") pod "2469fa44-5bee-4117-88f7-658a8ee48d6a" (UID: "2469fa44-5bee-4117-88f7-658a8ee48d6a"). InnerVolumeSpecName "kube-api-access-zr9vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:24:22.063557 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.063532    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-dshm" (OuterVolumeSpecName: "dshm") pod "2469fa44-5bee-4117-88f7-658a8ee48d6a" (UID: "2469fa44-5bee-4117-88f7-658a8ee48d6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:24:22.162366 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.162330    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2469fa44-5bee-4117-88f7-658a8ee48d6a-tls-certs\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:24:22.162516 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.162373    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr9vs\" (UniqueName: \"kubernetes.io/projected/2469fa44-5bee-4117-88f7-658a8ee48d6a-kube-api-access-zr9vs\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:24:22.162516 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.162397    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2469fa44-5bee-4117-88f7-658a8ee48d6a-dshm\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:24:22.359898 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.359809    2578 generic.go:356] "Generic (PLEG): container finished" podID="2469fa44-5bee-4117-88f7-658a8ee48d6a" containerID="2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd" exitCode=0
Jul 28 21:24:22.359898 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.359851    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"
Jul 28 21:24:22.360168 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.359896    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch" event={"ID":"2469fa44-5bee-4117-88f7-658a8ee48d6a","Type":"ContainerDied","Data":"2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd"}
Jul 28 21:24:22.360168 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.359938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch" event={"ID":"2469fa44-5bee-4117-88f7-658a8ee48d6a","Type":"ContainerDied","Data":"5067abe6c68698c496305fede8c38d60c2e6ac883ecba6946cb17a7db0950563"}
Jul 28 21:24:22.360168 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.359957    2578 scope.go:117] "RemoveContainer" containerID="2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd"
Jul 28 21:24:22.374386 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.374367    2578 scope.go:117] "RemoveContainer" containerID="2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd"
Jul 28 21:24:22.374646 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:24:22.374627    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd\": container with ID starting with 2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd not found: ID does not exist" containerID="2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd"
Jul 28 21:24:22.374691 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.374656    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd"} err="failed to get container status \"2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd\": rpc error: code = NotFound desc = could not find container \"2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd\": container with ID starting with 2d8ca8c1cb98bd187eaccd65cdace1459d9ff2fc068aec89a8ce532c8c02e3bd not found: ID does not exist"
Jul 28 21:24:22.387264 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.387229    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"]
Jul 28 21:24:22.392130 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:22.392102    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9bf2ee/test-model-9bf2ee-kserve-76c6b4cdc5-khjch"]
Jul 28 21:24:23.529606 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:23.529571    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2469fa44-5bee-4117-88f7-658a8ee48d6a" path="/var/lib/kubelet/pods/2469fa44-5bee-4117-88f7-658a8ee48d6a/volumes"
Jul 28 21:24:35.771158 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:35.771121    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"]
Jul 28 21:24:35.771576 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:35.771397    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw" podUID="bf6f9721-3b48-428a-898a-1610fc742318" containerName="payload-processing" containerID="cri-o://0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac" gracePeriod=30
Jul 28 21:24:35.777185 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:35.777158    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"]
Jul 28 21:24:35.777496 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:35.777462    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w" podUID="6bc355e9-8726-400c-827c-7b9d54d70895" containerName="payload-pre-processing" containerID="cri-o://aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31" gracePeriod=30
Jul 28 21:24:36.035625 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.035563    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:24:36.060643 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.060621    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:24:36.199756 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.199726    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk48j\" (UniqueName: \"kubernetes.io/projected/bf6f9721-3b48-428a-898a-1610fc742318-kube-api-access-zk48j\") pod \"bf6f9721-3b48-428a-898a-1610fc742318\" (UID: \"bf6f9721-3b48-428a-898a-1610fc742318\") "
Jul 28 21:24:36.199917 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.199791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq2h4\" (UniqueName: \"kubernetes.io/projected/6bc355e9-8726-400c-827c-7b9d54d70895-kube-api-access-dq2h4\") pod \"6bc355e9-8726-400c-827c-7b9d54d70895\" (UID: \"6bc355e9-8726-400c-827c-7b9d54d70895\") "
Jul 28 21:24:36.199917 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.199826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc355e9-8726-400c-827c-7b9d54d70895-plugins-config-volume\") pod \"6bc355e9-8726-400c-827c-7b9d54d70895\" (UID: \"6bc355e9-8726-400c-827c-7b9d54d70895\") "
Jul 28 21:24:36.199917 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.199863    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6f9721-3b48-428a-898a-1610fc742318-plugins-config-volume\") pod \"bf6f9721-3b48-428a-898a-1610fc742318\" (UID: \"bf6f9721-3b48-428a-898a-1610fc742318\") "
Jul 28 21:24:36.200329 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.200300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc355e9-8726-400c-827c-7b9d54d70895-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6bc355e9-8726-400c-827c-7b9d54d70895" (UID: "6bc355e9-8726-400c-827c-7b9d54d70895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:24:36.200329 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.200313    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6f9721-3b48-428a-898a-1610fc742318-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf6f9721-3b48-428a-898a-1610fc742318" (UID: "bf6f9721-3b48-428a-898a-1610fc742318"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:24:36.202051 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.202028    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6f9721-3b48-428a-898a-1610fc742318-kube-api-access-zk48j" (OuterVolumeSpecName: "kube-api-access-zk48j") pod "bf6f9721-3b48-428a-898a-1610fc742318" (UID: "bf6f9721-3b48-428a-898a-1610fc742318"). InnerVolumeSpecName "kube-api-access-zk48j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:24:36.202166 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.202142    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc355e9-8726-400c-827c-7b9d54d70895-kube-api-access-dq2h4" (OuterVolumeSpecName: "kube-api-access-dq2h4") pod "6bc355e9-8726-400c-827c-7b9d54d70895" (UID: "6bc355e9-8726-400c-827c-7b9d54d70895"). InnerVolumeSpecName "kube-api-access-dq2h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:24:36.300663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.300587    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6f9721-3b48-428a-898a-1610fc742318-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:24:36.300663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.300613    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk48j\" (UniqueName: \"kubernetes.io/projected/bf6f9721-3b48-428a-898a-1610fc742318-kube-api-access-zk48j\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:24:36.300663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.300622    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq2h4\" (UniqueName: \"kubernetes.io/projected/6bc355e9-8726-400c-827c-7b9d54d70895-kube-api-access-dq2h4\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:24:36.300663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.300631    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc355e9-8726-400c-827c-7b9d54d70895-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:24:36.430502 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.430469    2578 generic.go:356] "Generic (PLEG): container finished" podID="6bc355e9-8726-400c-827c-7b9d54d70895" containerID="aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31" exitCode=0
Jul 28 21:24:36.430667 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.430508    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"
Jul 28 21:24:36.430667 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.430555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w" event={"ID":"6bc355e9-8726-400c-827c-7b9d54d70895","Type":"ContainerDied","Data":"aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31"}
Jul 28 21:24:36.430667 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.430614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w" event={"ID":"6bc355e9-8726-400c-827c-7b9d54d70895","Type":"ContainerDied","Data":"af767e9535239025c8dfa269bc93790836061d5d1bc96a957caa307680409f41"}
Jul 28 21:24:36.430667 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.430642    2578 scope.go:117] "RemoveContainer" containerID="aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31"
Jul 28 21:24:36.431685 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.431628    2578 generic.go:356] "Generic (PLEG): container finished" podID="bf6f9721-3b48-428a-898a-1610fc742318" containerID="0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac" exitCode=0
Jul 28 21:24:36.431685 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.431667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw" event={"ID":"bf6f9721-3b48-428a-898a-1610fc742318","Type":"ContainerDied","Data":"0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac"}
Jul 28 21:24:36.431794 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.431686    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"
Jul 28 21:24:36.431794 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.431710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw" event={"ID":"bf6f9721-3b48-428a-898a-1610fc742318","Type":"ContainerDied","Data":"f01051b9af63a83b22546def6e648c743110fb3e0bd8b526ac8df69fafcc7fce"}
Jul 28 21:24:36.439262 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.439244    2578 scope.go:117] "RemoveContainer" containerID="aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31"
Jul 28 21:24:36.439535 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:24:36.439518    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31\": container with ID starting with aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31 not found: ID does not exist" containerID="aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31"
Jul 28 21:24:36.439600 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.439542    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31"} err="failed to get container status \"aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31\": rpc error: code = NotFound desc = could not find container \"aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31\": container with ID starting with aa776df6e243d8124416cad6e5ce816e793cba3964132def87986930e3d7ed31 not found: ID does not exist"
Jul 28 21:24:36.439600 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.439559    2578 scope.go:117] "RemoveContainer" containerID="0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac"
Jul 28 21:24:36.450850 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.450832    2578 scope.go:117] "RemoveContainer" containerID="0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac"
Jul 28 21:24:36.451129 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:24:36.451108    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac\": container with ID starting with 0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac not found: ID does not exist" containerID="0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac"
Jul 28 21:24:36.451191 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.451136    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac"} err="failed to get container status \"0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac\": rpc error: code = NotFound desc = could not find container \"0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac\": container with ID starting with 0bd42b3afbcba122c4ad199b3a87df667b4fca76a85f4f80af52cbc1fe03f9ac not found: ID does not exist"
Jul 28 21:24:36.455338 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.455317    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"]
Jul 28 21:24:36.459271 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.459251    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9bf2ee-6b64bfc65b-fd86w"]
Jul 28 21:24:36.469008 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.468984    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"]
Jul 28 21:24:36.473606 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:36.473587    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bf2ee-5b8f576dcd-78pvw"]
Jul 28 21:24:37.528890 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:37.528861    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc355e9-8726-400c-827c-7b9d54d70895" path="/var/lib/kubelet/pods/6bc355e9-8726-400c-827c-7b9d54d70895/volumes"
Jul 28 21:24:37.529281 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:37.529229    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf6f9721-3b48-428a-898a-1610fc742318" path="/var/lib/kubelet/pods/bf6f9721-3b48-428a-898a-1610fc742318/volumes"
Jul 28 21:24:47.464639 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.464596    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"]
Jul 28 21:24:47.465266 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.465212    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2469fa44-5bee-4117-88f7-658a8ee48d6a" containerName="main"
Jul 28 21:24:47.465266 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.465232    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2469fa44-5bee-4117-88f7-658a8ee48d6a" containerName="main"
Jul 28 21:24:47.465266 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.465250    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf6f9721-3b48-428a-898a-1610fc742318" containerName="payload-processing"
Jul 28 21:24:47.465266 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.465258    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf6f9721-3b48-428a-898a-1610fc742318" containerName="payload-processing"
Jul 28 21:24:47.465486 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.465272    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc355e9-8726-400c-827c-7b9d54d70895" containerName="payload-pre-processing"
Jul 28 21:24:47.465486 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.465280    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc355e9-8726-400c-827c-7b9d54d70895" containerName="payload-pre-processing"
Jul 28 21:24:47.465486 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.465369    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2469fa44-5bee-4117-88f7-658a8ee48d6a" containerName="main"
Jul 28 21:24:47.465486 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.465380    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc355e9-8726-400c-827c-7b9d54d70895" containerName="payload-pre-processing"
Jul 28 21:24:47.465486 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.465390    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf6f9721-3b48-428a-898a-1610fc742318" containerName="payload-processing"
Jul 28 21:24:47.468398 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.468378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:24:47.471004 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.470966    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ee8397\""
Jul 28 21:24:47.471181 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.471033    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ee8397-dockercfg-47xmr\""
Jul 28 21:24:47.478515 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.478494    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"]
Jul 28 21:24:47.491491 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.491459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cca2c47-bbe1-4687-912c-9e69920c187c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227\" (UID: \"2cca2c47-bbe1-4687-912c-9e69920c187c\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:24:47.491642 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.491531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6rqt\" (UniqueName: \"kubernetes.io/projected/2cca2c47-bbe1-4687-912c-9e69920c187c-kube-api-access-d6rqt\") pod \"payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227\" (UID: \"2cca2c47-bbe1-4687-912c-9e69920c187c\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:24:47.591953 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.591912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6rqt\" (UniqueName: \"kubernetes.io/projected/2cca2c47-bbe1-4687-912c-9e69920c187c-kube-api-access-d6rqt\") pod \"payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227\" (UID: \"2cca2c47-bbe1-4687-912c-9e69920c187c\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:24:47.592185 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.592022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cca2c47-bbe1-4687-912c-9e69920c187c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227\" (UID: \"2cca2c47-bbe1-4687-912c-9e69920c187c\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:24:47.592744 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.592721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cca2c47-bbe1-4687-912c-9e69920c187c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227\" (UID: \"2cca2c47-bbe1-4687-912c-9e69920c187c\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:24:47.599876 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.599852    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6rqt\" (UniqueName: \"kubernetes.io/projected/2cca2c47-bbe1-4687-912c-9e69920c187c-kube-api-access-d6rqt\") pod \"payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227\" (UID: \"2cca2c47-bbe1-4687-912c-9e69920c187c\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:24:47.780555 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.780519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:24:47.919048 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:47.918842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"]
Jul 28 21:24:47.919222 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:24:47.919136    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cca2c47_bbe1_4687_912c_9e69920c187c.slice/crio-c95cbdf288fd3b32449b899432072602502eded6055b2a69c62fefbf9375a041 WatchSource:0}: Error finding container c95cbdf288fd3b32449b899432072602502eded6055b2a69c62fefbf9375a041: Status 404 returned error can't find the container with id c95cbdf288fd3b32449b899432072602502eded6055b2a69c62fefbf9375a041
Jul 28 21:24:48.496747 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:48.496707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227" event={"ID":"2cca2c47-bbe1-4687-912c-9e69920c187c","Type":"ContainerStarted","Data":"1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214"}
Jul 28 21:24:48.497172 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:48.496778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227" event={"ID":"2cca2c47-bbe1-4687-912c-9e69920c187c","Type":"ContainerStarted","Data":"c95cbdf288fd3b32449b899432072602502eded6055b2a69c62fefbf9375a041"}
Jul 28 21:24:48.497172 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:48.496823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:24:48.513754 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:48.513712    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227" podStartSLOduration=1.5136995199999999 podStartE2EDuration="1.51369952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:24:48.511775897 +0000 UTC m=+2377.366619835" watchObservedRunningTime="2026-07-28 21:24:48.51369952 +0000 UTC m=+2377.368543460"
Jul 28 21:24:59.504011 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:24:59.503982    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:25:59.326438 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.326403    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"]
Jul 28 21:25:59.327089 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.326663    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227" podUID="2cca2c47-bbe1-4687-912c-9e69920c187c" containerName="payload-processing" containerID="cri-o://1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214" gracePeriod=30
Jul 28 21:25:59.502509 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.502462    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227" podUID="2cca2c47-bbe1-4687-912c-9e69920c187c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.51:9004: connect: connection refused"
Jul 28 21:25:59.589700 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.589635    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:25:59.645351 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.645319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cca2c47-bbe1-4687-912c-9e69920c187c-plugins-config-volume\") pod \"2cca2c47-bbe1-4687-912c-9e69920c187c\" (UID: \"2cca2c47-bbe1-4687-912c-9e69920c187c\") "
Jul 28 21:25:59.645543 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.645447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6rqt\" (UniqueName: \"kubernetes.io/projected/2cca2c47-bbe1-4687-912c-9e69920c187c-kube-api-access-d6rqt\") pod \"2cca2c47-bbe1-4687-912c-9e69920c187c\" (UID: \"2cca2c47-bbe1-4687-912c-9e69920c187c\") "
Jul 28 21:25:59.645798 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.645763    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cca2c47-bbe1-4687-912c-9e69920c187c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2cca2c47-bbe1-4687-912c-9e69920c187c" (UID: "2cca2c47-bbe1-4687-912c-9e69920c187c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:25:59.647787 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.647757    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cca2c47-bbe1-4687-912c-9e69920c187c-kube-api-access-d6rqt" (OuterVolumeSpecName: "kube-api-access-d6rqt") pod "2cca2c47-bbe1-4687-912c-9e69920c187c" (UID: "2cca2c47-bbe1-4687-912c-9e69920c187c"). InnerVolumeSpecName "kube-api-access-d6rqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:59.746937 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.746905    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cca2c47-bbe1-4687-912c-9e69920c187c-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:25:59.746937 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.746935    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6rqt\" (UniqueName: \"kubernetes.io/projected/2cca2c47-bbe1-4687-912c-9e69920c187c-kube-api-access-d6rqt\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:25:59.854412 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.854324    2578 generic.go:356] "Generic (PLEG): container finished" podID="2cca2c47-bbe1-4687-912c-9e69920c187c" containerID="1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214" exitCode=0
Jul 28 21:25:59.854412 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.854369    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"
Jul 28 21:25:59.854412 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.854399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227" event={"ID":"2cca2c47-bbe1-4687-912c-9e69920c187c","Type":"ContainerDied","Data":"1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214"}
Jul 28 21:25:59.854679 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.854442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227" event={"ID":"2cca2c47-bbe1-4687-912c-9e69920c187c","Type":"ContainerDied","Data":"c95cbdf288fd3b32449b899432072602502eded6055b2a69c62fefbf9375a041"}
Jul 28 21:25:59.854679 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.854465    2578 scope.go:117] "RemoveContainer" containerID="1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214"
Jul 28 21:25:59.864016 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.863999    2578 scope.go:117] "RemoveContainer" containerID="1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214"
Jul 28 21:25:59.864317 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:25:59.864298    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214\": container with ID starting with 1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214 not found: ID does not exist" containerID="1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214"
Jul 28 21:25:59.864396 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.864325    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214"} err="failed to get container status \"1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214\": rpc error: code = NotFound desc = could not find container \"1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214\": container with ID starting with 1936d3e84897531381b9126bdd0f55a095c3ac86ed6e62aa3c33a9f5f4723214 not found: ID does not exist"
Jul 28 21:25:59.878302 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.878275    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"]
Jul 28 21:25:59.884420 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:25:59.884399    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8397-6d5f96c898-gw227"]
Jul 28 21:26:01.529453 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:26:01.529423    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cca2c47-bbe1-4687-912c-9e69920c187c" path="/var/lib/kubelet/pods/2cca2c47-bbe1-4687-912c-9e69920c187c/volumes"
Jul 28 21:27:02.923733 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.923638    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"]
Jul 28 21:27:02.924268 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.924229    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cca2c47-bbe1-4687-912c-9e69920c187c" containerName="payload-processing"
Jul 28 21:27:02.924268 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.924250    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cca2c47-bbe1-4687-912c-9e69920c187c" containerName="payload-processing"
Jul 28 21:27:02.924393 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.924346    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cca2c47-bbe1-4687-912c-9e69920c187c" containerName="payload-processing"
Jul 28 21:27:02.927466 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.927444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:27:02.929854 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.929833    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-7076b1\""
Jul 28 21:27:02.929854 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.929842    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-7076b1-dockercfg-hw56j\""
Jul 28 21:27:02.938460 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.938428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"]
Jul 28 21:27:02.954593 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.954557    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"]
Jul 28 21:27:02.958419 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.958398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:27:02.976545 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:02.976510    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"]
Jul 28 21:27:03.027249 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.027213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jt2p\" (UniqueName: \"kubernetes.io/projected/f975d585-01de-46e3-be1e-5f3acfb326b2-kube-api-access-7jt2p\") pod \"payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd\" (UID: \"f975d585-01de-46e3-be1e-5f3acfb326b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:27:03.027249 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.027250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f975d585-01de-46e3-be1e-5f3acfb326b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd\" (UID: \"f975d585-01de-46e3-be1e-5f3acfb326b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:27:03.127993 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.127941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92b6\" (UniqueName: \"kubernetes.io/projected/c168ec44-10dc-4b12-a22b-3394de38b689-kube-api-access-j92b6\") pod \"payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt\" (UID: \"c168ec44-10dc-4b12-a22b-3394de38b689\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:27:03.128236 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.128015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jt2p\" (UniqueName: \"kubernetes.io/projected/f975d585-01de-46e3-be1e-5f3acfb326b2-kube-api-access-7jt2p\") pod \"payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd\" (UID: \"f975d585-01de-46e3-be1e-5f3acfb326b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:27:03.128236 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.128121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f975d585-01de-46e3-be1e-5f3acfb326b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd\" (UID: \"f975d585-01de-46e3-be1e-5f3acfb326b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:27:03.128236 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.128210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c168ec44-10dc-4b12-a22b-3394de38b689-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt\" (UID: \"c168ec44-10dc-4b12-a22b-3394de38b689\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:27:03.128802 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.128784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f975d585-01de-46e3-be1e-5f3acfb326b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd\" (UID: \"f975d585-01de-46e3-be1e-5f3acfb326b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:27:03.136703 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.136678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jt2p\" (UniqueName: \"kubernetes.io/projected/f975d585-01de-46e3-be1e-5f3acfb326b2-kube-api-access-7jt2p\") pod \"payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd\" (UID: \"f975d585-01de-46e3-be1e-5f3acfb326b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:27:03.229434 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.229403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c168ec44-10dc-4b12-a22b-3394de38b689-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt\" (UID: \"c168ec44-10dc-4b12-a22b-3394de38b689\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:27:03.229635 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.229475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92b6\" (UniqueName: \"kubernetes.io/projected/c168ec44-10dc-4b12-a22b-3394de38b689-kube-api-access-j92b6\") pod \"payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt\" (UID: \"c168ec44-10dc-4b12-a22b-3394de38b689\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:27:03.230200 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.230176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c168ec44-10dc-4b12-a22b-3394de38b689-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt\" (UID: \"c168ec44-10dc-4b12-a22b-3394de38b689\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:27:03.238050 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.238026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92b6\" (UniqueName: \"kubernetes.io/projected/c168ec44-10dc-4b12-a22b-3394de38b689-kube-api-access-j92b6\") pod \"payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt\" (UID: \"c168ec44-10dc-4b12-a22b-3394de38b689\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:27:03.238228 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.238209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:27:03.268920 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.268886    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:27:03.395824 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.395792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"]
Jul 28 21:27:03.397987 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:27:03.397962    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf975d585_01de_46e3_be1e_5f3acfb326b2.slice/crio-7d33b826b2adfb4cfa5abb461b9a723b13e272945c0803da2d05cf6e1acd000a WatchSource:0}: Error finding container 7d33b826b2adfb4cfa5abb461b9a723b13e272945c0803da2d05cf6e1acd000a: Status 404 returned error can't find the container with id 7d33b826b2adfb4cfa5abb461b9a723b13e272945c0803da2d05cf6e1acd000a
Jul 28 21:27:03.422119 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:03.422063    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"]
Jul 28 21:27:03.423979 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:27:03.423953    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc168ec44_10dc_4b12_a22b_3394de38b689.slice/crio-eae7b0d74ad16c8d6b90e09662fe0bbd6a3f19b5eb89d266bdb3ed934dfac1e1 WatchSource:0}: Error finding container eae7b0d74ad16c8d6b90e09662fe0bbd6a3f19b5eb89d266bdb3ed934dfac1e1: Status 404 returned error can't find the container with id eae7b0d74ad16c8d6b90e09662fe0bbd6a3f19b5eb89d266bdb3ed934dfac1e1
Jul 28 21:27:04.164655 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:04.164610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt" event={"ID":"c168ec44-10dc-4b12-a22b-3394de38b689","Type":"ContainerStarted","Data":"657efbd2cb7e84dfa79662ef9f63e28f86574884e3ec5325b270521a5963d9aa"}
Jul 28 21:27:04.165128 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:04.164677    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:27:04.165128 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:04.164701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt" event={"ID":"c168ec44-10dc-4b12-a22b-3394de38b689","Type":"ContainerStarted","Data":"eae7b0d74ad16c8d6b90e09662fe0bbd6a3f19b5eb89d266bdb3ed934dfac1e1"}
Jul 28 21:27:04.172400 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:04.166933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd" event={"ID":"f975d585-01de-46e3-be1e-5f3acfb326b2","Type":"ContainerStarted","Data":"8ad76d85c0635831cb7ad65ee27374562c117a2b0531e2f111056753fd2c74c7"}
Jul 28 21:27:04.172400 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:04.167034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd" event={"ID":"f975d585-01de-46e3-be1e-5f3acfb326b2","Type":"ContainerStarted","Data":"7d33b826b2adfb4cfa5abb461b9a723b13e272945c0803da2d05cf6e1acd000a"}
Jul 28 21:27:04.172400 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:04.167253    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:27:04.182392 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:04.182340    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt" podStartSLOduration=2.182320105 podStartE2EDuration="2.182320105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:27:04.180515624 +0000 UTC m=+2513.035359534" watchObservedRunningTime="2026-07-28 21:27:04.182320105 +0000 UTC m=+2513.037164018"
Jul 28 21:27:04.198694 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:04.198637    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd" podStartSLOduration=2.198619283 podStartE2EDuration="2.198619283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:27:04.198497447 +0000 UTC m=+2513.053341356" watchObservedRunningTime="2026-07-28 21:27:04.198619283 +0000 UTC m=+2513.053463195"
Jul 28 21:27:15.174884 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:15.174837    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:27:15.175445 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:15.174945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:27:40.433624 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.433592    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"]
Jul 28 21:27:40.437389 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.437369    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.439927 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.439903    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b439d9\"/\"kube-root-ca.crt\""
Jul 28 21:27:40.439927 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.439920    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b439d9\"/\"openshift-service-ca.crt\""
Jul 28 21:27:40.440953 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.440935    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b439d9\"/\"default-dockercfg-jh228\""
Jul 28 21:27:40.441017 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.440957    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b439d9\"/\"auth-test-model-b439d9-kserve-self-signed-certs\""
Jul 28 21:27:40.447944 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.447919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"]
Jul 28 21:27:40.581409 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.581363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tmp-dir\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.581619 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.581421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wtt2\" (UniqueName: \"kubernetes.io/projected/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-kube-api-access-6wtt2\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.581619 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.581494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-model-cache\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.581619 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.581529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-home\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.581619 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.581610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-dshm\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.581779 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.581644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tls-certs\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.682352 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.682301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-home\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.682565 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.682375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-dshm\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.682565 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.682393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tls-certs\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.682565 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.682458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tmp-dir\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.682565 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.682488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wtt2\" (UniqueName: \"kubernetes.io/projected/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-kube-api-access-6wtt2\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.682565 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.682513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-model-cache\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.683230 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.682775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-home\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.683230 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.682870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-model-cache\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.683230 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.682929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tmp-dir\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.685384 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.685311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tls-certs\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.685384 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.685351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-dshm\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.691845 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.691819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wtt2\" (UniqueName: \"kubernetes.io/projected/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-kube-api-access-6wtt2\") pod \"auth-test-model-b439d9-kserve-7d5796f759-6w2j5\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") " pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.746949 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.746911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:40.886140 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:40.886114    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"]
Jul 28 21:27:40.888717 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:27:40.888678    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48d89ae4_648d_4cb9_9393_3d1ff3fa1ec6.slice/crio-d74a612ef0a50d5236f963510807a3ef96c8e8bfcf59002fad79972b8b05699f WatchSource:0}: Error finding container d74a612ef0a50d5236f963510807a3ef96c8e8bfcf59002fad79972b8b05699f: Status 404 returned error can't find the container with id d74a612ef0a50d5236f963510807a3ef96c8e8bfcf59002fad79972b8b05699f
Jul 28 21:27:41.351093 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:41.351050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5" event={"ID":"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6","Type":"ContainerStarted","Data":"f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e"}
Jul 28 21:27:41.351258 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:41.351120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5" event={"ID":"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6","Type":"ContainerStarted","Data":"d74a612ef0a50d5236f963510807a3ef96c8e8bfcf59002fad79972b8b05699f"}
Jul 28 21:27:41.369401 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:41.369350    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5" podStartSLOduration=1.369313014 podStartE2EDuration="1.369313014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:27:41.367904828 +0000 UTC m=+2550.222748739" watchObservedRunningTime="2026-07-28 21:27:41.369313014 +0000 UTC m=+2550.224156925"
Jul 28 21:27:50.747855 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:50.747811    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:50.748435 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:50.747868    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:50.760760 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:50.760734    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:27:51.414577 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:27:51.414547    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:28:22.846269 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.846234    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"]
Jul 28 21:28:22.850313 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.850293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:22.852891 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.852869    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7076b1\"/\"openshift-service-ca.crt\""
Jul 28 21:28:22.854127 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.853877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7076b1\"/\"auth-test-model-7076b1-kserve-self-signed-certs\""
Jul 28 21:28:22.854127 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.853915    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7076b1\"/\"default-dockercfg-84l9t\""
Jul 28 21:28:22.854127 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.853920    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7076b1\"/\"kube-root-ca.crt\""
Jul 28 21:28:22.857637 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.857236    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"]
Jul 28 21:28:22.989496 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.989461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9pp8\" (UniqueName: \"kubernetes.io/projected/a7661111-01b2-4c98-a0d7-196b5e8e1208-kube-api-access-k9pp8\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:22.989706 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.989504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-model-cache\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:22.989706 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.989588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-dshm\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:22.989706 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.989625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7661111-01b2-4c98-a0d7-196b5e8e1208-tls-certs\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:22.989706 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.989655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-home\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:22.989706 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:22.989683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-tmp-dir\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.090875 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.090829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-model-cache\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.091108 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.090902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-dshm\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.091108 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.090922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7661111-01b2-4c98-a0d7-196b5e8e1208-tls-certs\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.091108 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.090943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-home\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.091108 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.090966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-tmp-dir\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.091108 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.091018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9pp8\" (UniqueName: \"kubernetes.io/projected/a7661111-01b2-4c98-a0d7-196b5e8e1208-kube-api-access-k9pp8\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.091392 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.091345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-model-cache\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.091392 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.091377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-home\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.091504 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.091401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-tmp-dir\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.093466 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.093439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-dshm\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.093581 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.093467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7661111-01b2-4c98-a0d7-196b5e8e1208-tls-certs\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.099657 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.099599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9pp8\" (UniqueName: \"kubernetes.io/projected/a7661111-01b2-4c98-a0d7-196b5e8e1208-kube-api-access-k9pp8\") pod \"auth-test-model-7076b1-kserve-7d478f5b9f-mwljr\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") " pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.160602 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.160569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:23.293376 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.293348    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"]
Jul 28 21:28:23.294782 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:28:23.294755    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7661111_01b2_4c98_a0d7_196b5e8e1208.slice/crio-6d64196771d22c954894a37bedf4c26531872a1b5ba075972d8c7708cfa07c5a WatchSource:0}: Error finding container 6d64196771d22c954894a37bedf4c26531872a1b5ba075972d8c7708cfa07c5a: Status 404 returned error can't find the container with id 6d64196771d22c954894a37bedf4c26531872a1b5ba075972d8c7708cfa07c5a
Jul 28 21:28:23.557365 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.557323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr" event={"ID":"a7661111-01b2-4c98-a0d7-196b5e8e1208","Type":"ContainerStarted","Data":"10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e"}
Jul 28 21:28:23.557365 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.557369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr" event={"ID":"a7661111-01b2-4c98-a0d7-196b5e8e1208","Type":"ContainerStarted","Data":"6d64196771d22c954894a37bedf4c26531872a1b5ba075972d8c7708cfa07c5a"}
Jul 28 21:28:23.575737 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:23.575689    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr" podStartSLOduration=1.575672671 podStartE2EDuration="1.575672671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:28:23.575043569 +0000 UTC m=+2592.429887493" watchObservedRunningTime="2026-07-28 21:28:23.575672671 +0000 UTC m=+2592.430516587"
Jul 28 21:28:33.161324 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:33.161244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:33.161324 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:33.161297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:33.173913 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:33.173890    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:28:33.618225 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:28:33.618194    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:32:36.952219 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:36.952180    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"]
Jul 28 21:32:37.824254 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:37.824223    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"]
Jul 28 21:32:51.971425 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:51.971383    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5" podUID="48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" containerName="main" containerID="cri-o://f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e" gracePeriod=45
Jul 28 21:32:52.228280 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.228223    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:32:52.359043 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tmp-dir\") pod \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") "
Jul 28 21:32:52.359225 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359129    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-home\") pod \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") "
Jul 28 21:32:52.359225 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359172    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tls-certs\") pod \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") "
Jul 28 21:32:52.359345 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359252    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-model-cache\") pod \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") "
Jul 28 21:32:52.359345 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359277    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wtt2\" (UniqueName: \"kubernetes.io/projected/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-kube-api-access-6wtt2\") pod \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") "
Jul 28 21:32:52.359345 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359306    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" (UID: "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:32:52.359503 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359383    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-dshm\") pod \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\" (UID: \"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6\") "
Jul 28 21:32:52.359503 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-home" (OuterVolumeSpecName: "home") pod "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" (UID: "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:32:52.359503 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359471    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-model-cache" (OuterVolumeSpecName: "model-cache") pod "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" (UID: "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:32:52.359742 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359718    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-model-cache\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:52.359742 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359744    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tmp-dir\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:52.360124 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.359756    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-home\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:52.361571 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.361549    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-dshm" (OuterVolumeSpecName: "dshm") pod "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" (UID: "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:32:52.361571 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.361548    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-kube-api-access-6wtt2" (OuterVolumeSpecName: "kube-api-access-6wtt2") pod "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" (UID: "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6"). InnerVolumeSpecName "kube-api-access-6wtt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:32:52.361722 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.361598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" (UID: "48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:32:52.461036 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.461004    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wtt2\" (UniqueName: \"kubernetes.io/projected/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-kube-api-access-6wtt2\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:52.461036 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.461031    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-dshm\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:52.461036 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.461041    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6-tls-certs\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:52.843635 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.843571    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr" podUID="a7661111-01b2-4c98-a0d7-196b5e8e1208" containerName="main" containerID="cri-o://10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e" gracePeriod=45
Jul 28 21:32:52.876892 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.876866    2578 generic.go:356] "Generic (PLEG): container finished" podID="48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" containerID="f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e" exitCode=0
Jul 28 21:32:52.877044 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.876907    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"
Jul 28 21:32:52.877044 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.876960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5" event={"ID":"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6","Type":"ContainerDied","Data":"f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e"}
Jul 28 21:32:52.877044 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.877007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5" event={"ID":"48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6","Type":"ContainerDied","Data":"d74a612ef0a50d5236f963510807a3ef96c8e8bfcf59002fad79972b8b05699f"}
Jul 28 21:32:52.877044 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.877031    2578 scope.go:117] "RemoveContainer" containerID="f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e"
Jul 28 21:32:52.914599 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.914576    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"]
Jul 28 21:32:52.918563 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.918542    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/auth-test-model-b439d9-kserve-7d5796f759-6w2j5"]
Jul 28 21:32:52.955352 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.955333    2578 scope.go:117] "RemoveContainer" containerID="f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e"
Jul 28 21:32:52.955732 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:32:52.955712    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e\": container with ID starting with f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e not found: ID does not exist" containerID="f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e"
Jul 28 21:32:52.955798 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:52.955746    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e"} err="failed to get container status \"f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e\": rpc error: code = NotFound desc = could not find container \"f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e\": container with ID starting with f033be96524492dc4418ba3c0676da2bd3b70ef13e6fcd37baf9fe62262e2f5e not found: ID does not exist"
Jul 28 21:32:53.102836 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.102778    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:32:53.167798 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.167762    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7661111-01b2-4c98-a0d7-196b5e8e1208-tls-certs\") pod \"a7661111-01b2-4c98-a0d7-196b5e8e1208\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") "
Jul 28 21:32:53.167950 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.167845    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-home\") pod \"a7661111-01b2-4c98-a0d7-196b5e8e1208\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") "
Jul 28 21:32:53.167950 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.167888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-dshm\") pod \"a7661111-01b2-4c98-a0d7-196b5e8e1208\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") "
Jul 28 21:32:53.167950 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.167934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-model-cache\") pod \"a7661111-01b2-4c98-a0d7-196b5e8e1208\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") "
Jul 28 21:32:53.168193 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.167984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9pp8\" (UniqueName: \"kubernetes.io/projected/a7661111-01b2-4c98-a0d7-196b5e8e1208-kube-api-access-k9pp8\") pod \"a7661111-01b2-4c98-a0d7-196b5e8e1208\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") "
Jul 28 21:32:53.168193 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.168027    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-tmp-dir\") pod \"a7661111-01b2-4c98-a0d7-196b5e8e1208\" (UID: \"a7661111-01b2-4c98-a0d7-196b5e8e1208\") "
Jul 28 21:32:53.168193 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.168137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-home" (OuterVolumeSpecName: "home") pod "a7661111-01b2-4c98-a0d7-196b5e8e1208" (UID: "a7661111-01b2-4c98-a0d7-196b5e8e1208"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:32:53.168364 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.168263    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7661111-01b2-4c98-a0d7-196b5e8e1208" (UID: "a7661111-01b2-4c98-a0d7-196b5e8e1208"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:32:53.168409 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.168367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7661111-01b2-4c98-a0d7-196b5e8e1208" (UID: "a7661111-01b2-4c98-a0d7-196b5e8e1208"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:32:53.168473 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.168457    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-home\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:53.168511 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.168480    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-model-cache\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:53.168511 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.168494    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-tmp-dir\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:53.170259 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.170236    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-dshm" (OuterVolumeSpecName: "dshm") pod "a7661111-01b2-4c98-a0d7-196b5e8e1208" (UID: "a7661111-01b2-4c98-a0d7-196b5e8e1208"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:32:53.170329 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.170236    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7661111-01b2-4c98-a0d7-196b5e8e1208-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7661111-01b2-4c98-a0d7-196b5e8e1208" (UID: "a7661111-01b2-4c98-a0d7-196b5e8e1208"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:32:53.170329 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.170260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7661111-01b2-4c98-a0d7-196b5e8e1208-kube-api-access-k9pp8" (OuterVolumeSpecName: "kube-api-access-k9pp8") pod "a7661111-01b2-4c98-a0d7-196b5e8e1208" (UID: "a7661111-01b2-4c98-a0d7-196b5e8e1208"). InnerVolumeSpecName "kube-api-access-k9pp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:32:53.269626 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.269597    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7661111-01b2-4c98-a0d7-196b5e8e1208-tls-certs\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:53.269626 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.269622    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7661111-01b2-4c98-a0d7-196b5e8e1208-dshm\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:53.269798 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.269632    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9pp8\" (UniqueName: \"kubernetes.io/projected/a7661111-01b2-4c98-a0d7-196b5e8e1208-kube-api-access-k9pp8\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:32:53.530436 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.530405    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" path="/var/lib/kubelet/pods/48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6/volumes"
Jul 28 21:32:53.883554 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.883460    2578 generic.go:356] "Generic (PLEG): container finished" podID="a7661111-01b2-4c98-a0d7-196b5e8e1208" containerID="10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e" exitCode=0
Jul 28 21:32:53.883554 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.883529    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"
Jul 28 21:32:53.883793 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.883550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr" event={"ID":"a7661111-01b2-4c98-a0d7-196b5e8e1208","Type":"ContainerDied","Data":"10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e"}
Jul 28 21:32:53.883793 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.883607    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr" event={"ID":"a7661111-01b2-4c98-a0d7-196b5e8e1208","Type":"ContainerDied","Data":"6d64196771d22c954894a37bedf4c26531872a1b5ba075972d8c7708cfa07c5a"}
Jul 28 21:32:53.883793 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.883630    2578 scope.go:117] "RemoveContainer" containerID="10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e"
Jul 28 21:32:53.892486 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.892466    2578 scope.go:117] "RemoveContainer" containerID="10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e"
Jul 28 21:32:53.892760 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:32:53.892744    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e\": container with ID starting with 10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e not found: ID does not exist" containerID="10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e"
Jul 28 21:32:53.892819 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.892768    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e"} err="failed to get container status \"10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e\": rpc error: code = NotFound desc = could not find container \"10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e\": container with ID starting with 10c27021fb295ec258b63874dbd78ee170056cd10e48f7a6229bf873a019f30e not found: ID does not exist"
Jul 28 21:32:53.903779 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.903759    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"]
Jul 28 21:32:53.907043 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:53.907022    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/auth-test-model-7076b1-kserve-7d478f5b9f-mwljr"]
Jul 28 21:32:55.531205 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:32:55.531175    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7661111-01b2-4c98-a0d7-196b5e8e1208" path="/var/lib/kubelet/pods/a7661111-01b2-4c98-a0d7-196b5e8e1208/volumes"
Jul 28 21:33:32.950054 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.950020    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"]
Jul 28 21:33:32.950490 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.950391    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" containerName="main"
Jul 28 21:33:32.950490 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.950403    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" containerName="main"
Jul 28 21:33:32.950490 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.950429    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7661111-01b2-4c98-a0d7-196b5e8e1208" containerName="main"
Jul 28 21:33:32.950490 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.950435    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7661111-01b2-4c98-a0d7-196b5e8e1208" containerName="main"
Jul 28 21:33:32.950490 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.950489    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7661111-01b2-4c98-a0d7-196b5e8e1208" containerName="main"
Jul 28 21:33:32.950650 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.950496    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d89ae4-648d-4cb9-9393-3d1ff3fa1ec6" containerName="main"
Jul 28 21:33:32.953543 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.953527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:32.956092 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.956053    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7076b1\"/\"openshift-service-ca.crt\""
Jul 28 21:33:32.957091 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.957052    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7076b1\"/\"sub-test-model-7076b1-kserve-self-signed-certs\""
Jul 28 21:33:32.957188 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.957055    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7076b1\"/\"default-dockercfg-84l9t\""
Jul 28 21:33:32.957188 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.957057    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7076b1\"/\"kube-root-ca.crt\""
Jul 28 21:33:32.961963 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:32.961941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"]
Jul 28 21:33:33.033913 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.033880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tmp-dir\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.033913 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.033912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-home\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.034160 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.033938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm5zg\" (UniqueName: \"kubernetes.io/projected/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-kube-api-access-pm5zg\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.034160 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.034039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tls-certs\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.034160 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.034106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-dshm\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.034160 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.034136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-model-cache\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.134610 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.134566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tls-certs\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.134789 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.134634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-dshm\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.134789 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.134655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-model-cache\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.134789 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.134679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tmp-dir\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.134789 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.134693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-home\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.134998 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.134821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm5zg\" (UniqueName: \"kubernetes.io/projected/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-kube-api-access-pm5zg\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.135164 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.135135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-model-cache\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.135267 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.135175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-home\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.135267 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.135220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tmp-dir\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.137006 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.136989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-dshm\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.137215 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.137199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tls-certs\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.143380 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.143351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm5zg\" (UniqueName: \"kubernetes.io/projected/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-kube-api-access-pm5zg\") pod \"sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") " pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.263490 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.263455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:33.408327 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:33.408295    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"]
Jul 28 21:33:33.411785 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:33:33.411756    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e8d51a5_f24d_48cf_a652_f43756f7b8a6.slice/crio-59555dc9626cb8fd7920741ec0db7310c736d1fca6ca1bea2d0d5fdae8a88263 WatchSource:0}: Error finding container 59555dc9626cb8fd7920741ec0db7310c736d1fca6ca1bea2d0d5fdae8a88263: Status 404 returned error can't find the container with id 59555dc9626cb8fd7920741ec0db7310c736d1fca6ca1bea2d0d5fdae8a88263
Jul 28 21:33:34.083741 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:34.083701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c" event={"ID":"9e8d51a5-f24d-48cf-a652-f43756f7b8a6","Type":"ContainerStarted","Data":"246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479"}
Jul 28 21:33:34.083741 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:34.083757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c" event={"ID":"9e8d51a5-f24d-48cf-a652-f43756f7b8a6","Type":"ContainerStarted","Data":"59555dc9626cb8fd7920741ec0db7310c736d1fca6ca1bea2d0d5fdae8a88263"}
Jul 28 21:33:34.101200 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:34.101149    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c" podStartSLOduration=2.101133359 podStartE2EDuration="2.101133359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:33:34.100612461 +0000 UTC m=+2902.955456372" watchObservedRunningTime="2026-07-28 21:33:34.101133359 +0000 UTC m=+2902.955977272"
Jul 28 21:33:43.264280 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:43.264236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:43.265112 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:43.264294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:43.276878 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:43.276855    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:33:44.152243 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:33:44.152203    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:34:33.438683 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:33.438602    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"]
Jul 28 21:34:48.458905 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.458834    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c" podUID="9e8d51a5-f24d-48cf-a652-f43756f7b8a6" containerName="main" containerID="cri-o://246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479" gracePeriod=45
Jul 28 21:34:48.717403 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.717340    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:34:48.774900 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.774867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tmp-dir\") pod \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") "
Jul 28 21:34:48.775045 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.774915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-home\") pod \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") "
Jul 28 21:34:48.775045 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.774970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tls-certs\") pod \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") "
Jul 28 21:34:48.775045 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.775018    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-model-cache\") pod \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") "
Jul 28 21:34:48.775045 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.775043    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-dshm\") pod \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") "
Jul 28 21:34:48.775260 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.775070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm5zg\" (UniqueName: \"kubernetes.io/projected/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-kube-api-access-pm5zg\") pod \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\" (UID: \"9e8d51a5-f24d-48cf-a652-f43756f7b8a6\") "
Jul 28 21:34:48.775260 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.775164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e8d51a5-f24d-48cf-a652-f43756f7b8a6" (UID: "9e8d51a5-f24d-48cf-a652-f43756f7b8a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:34:48.775260 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.775206    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-home" (OuterVolumeSpecName: "home") pod "9e8d51a5-f24d-48cf-a652-f43756f7b8a6" (UID: "9e8d51a5-f24d-48cf-a652-f43756f7b8a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:34:48.775390 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.775349    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e8d51a5-f24d-48cf-a652-f43756f7b8a6" (UID: "9e8d51a5-f24d-48cf-a652-f43756f7b8a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:34:48.775449 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.775398    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tmp-dir\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:34:48.775449 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.775413    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-home\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:34:48.777286 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.777263    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-dshm" (OuterVolumeSpecName: "dshm") pod "9e8d51a5-f24d-48cf-a652-f43756f7b8a6" (UID: "9e8d51a5-f24d-48cf-a652-f43756f7b8a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:34:48.777286 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.777272    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e8d51a5-f24d-48cf-a652-f43756f7b8a6" (UID: "9e8d51a5-f24d-48cf-a652-f43756f7b8a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:34:48.777437 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.777351    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-kube-api-access-pm5zg" (OuterVolumeSpecName: "kube-api-access-pm5zg") pod "9e8d51a5-f24d-48cf-a652-f43756f7b8a6" (UID: "9e8d51a5-f24d-48cf-a652-f43756f7b8a6"). InnerVolumeSpecName "kube-api-access-pm5zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:34:48.875926 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.875886    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-tls-certs\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:34:48.875926 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.875923    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-model-cache\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:34:48.875926 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.875932    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-dshm\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:34:48.875926 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:48.875940    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm5zg\" (UniqueName: \"kubernetes.io/projected/9e8d51a5-f24d-48cf-a652-f43756f7b8a6-kube-api-access-pm5zg\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:34:49.456317 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:49.456284    2578 generic.go:356] "Generic (PLEG): container finished" podID="9e8d51a5-f24d-48cf-a652-f43756f7b8a6" containerID="246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479" exitCode=0
Jul 28 21:34:49.456317 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:49.456323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c" event={"ID":"9e8d51a5-f24d-48cf-a652-f43756f7b8a6","Type":"ContainerDied","Data":"246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479"}
Jul 28 21:34:49.456554 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:49.456340    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"
Jul 28 21:34:49.456554 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:49.456360    2578 scope.go:117] "RemoveContainer" containerID="246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479"
Jul 28 21:34:49.456554 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:49.456350    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c" event={"ID":"9e8d51a5-f24d-48cf-a652-f43756f7b8a6","Type":"ContainerDied","Data":"59555dc9626cb8fd7920741ec0db7310c736d1fca6ca1bea2d0d5fdae8a88263"}
Jul 28 21:34:49.465448 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:49.465230    2578 scope.go:117] "RemoveContainer" containerID="246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479"
Jul 28 21:34:49.465706 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:34:49.465521    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479\": container with ID starting with 246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479 not found: ID does not exist" containerID="246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479"
Jul 28 21:34:49.465706 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:49.465549    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479"} err="failed to get container status \"246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479\": rpc error: code = NotFound desc = could not find container \"246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479\": container with ID starting with 246b4b1b9e949ca3dd3bd77c2666c4b4413e6ef86168af323818029770e02479 not found: ID does not exist"
Jul 28 21:34:49.480999 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:49.480977    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"]
Jul 28 21:34:49.485297 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:49.485271    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7076b1/sub-test-model-7076b1-kserve-5cb7bbfc69-vjx7c"]
Jul 28 21:34:49.530263 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:49.530235    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e8d51a5-f24d-48cf-a652-f43756f7b8a6" path="/var/lib/kubelet/pods/9e8d51a5-f24d-48cf-a652-f43756f7b8a6/volumes"
Jul 28 21:34:50.263995 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.263964    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"]
Jul 28 21:34:50.264374 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.264361    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8d51a5-f24d-48cf-a652-f43756f7b8a6" containerName="main"
Jul 28 21:34:50.264417 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.264376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8d51a5-f24d-48cf-a652-f43756f7b8a6" containerName="main"
Jul 28 21:34:50.264453 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.264448    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e8d51a5-f24d-48cf-a652-f43756f7b8a6" containerName="main"
Jul 28 21:34:50.268530 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.268513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.271165 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.271144    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b439d9\"/\"kube-root-ca.crt\""
Jul 28 21:34:50.272146 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.272126    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b439d9\"/\"openshift-service-ca.crt\""
Jul 28 21:34:50.272146 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.272130    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b439d9\"/\"rate-test-model-b439d9-kserve-self-signed-certs\""
Jul 28 21:34:50.272303 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.272245    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b439d9\"/\"default-dockercfg-jh228\""
Jul 28 21:34:50.278262 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.278240    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"]
Jul 28 21:34:50.391402 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.391358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n454d\" (UniqueName: \"kubernetes.io/projected/904d0185-10fc-4a44-a66c-7bfd6df09936-kube-api-access-n454d\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.391585 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.391417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904d0185-10fc-4a44-a66c-7bfd6df09936-tls-certs\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.391585 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.391481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-tmp-dir\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.391585 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.391548    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-home\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.391585 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.391575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-dshm\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.391742 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.391607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-model-cache\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.492276 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.492222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n454d\" (UniqueName: \"kubernetes.io/projected/904d0185-10fc-4a44-a66c-7bfd6df09936-kube-api-access-n454d\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.492276 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.492275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904d0185-10fc-4a44-a66c-7bfd6df09936-tls-certs\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.492807 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.492334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-tmp-dir\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.492807 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.492370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-home\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.492807 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.492393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-dshm\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.492807 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.492432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-model-cache\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.493026 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.492881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-model-cache\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.493026 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.492912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-tmp-dir\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.493281 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.493158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-home\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.496251 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.496226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904d0185-10fc-4a44-a66c-7bfd6df09936-tls-certs\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.496811 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.496786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-dshm\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.502103 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.502053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n454d\" (UniqueName: \"kubernetes.io/projected/904d0185-10fc-4a44-a66c-7bfd6df09936-kube-api-access-n454d\") pod \"rate-test-model-b439d9-kserve-86d9978fb7-pltds\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") " pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.578443 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.578358    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:34:50.720957 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:50.720926    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"]
Jul 28 21:34:50.723007 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:34:50.722981    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod904d0185_10fc_4a44_a66c_7bfd6df09936.slice/crio-a37854f069d0a3911b5c6550b67c0eb9b95aec72db57b4e6fdcab425fa0b44dd WatchSource:0}: Error finding container a37854f069d0a3911b5c6550b67c0eb9b95aec72db57b4e6fdcab425fa0b44dd: Status 404 returned error can't find the container with id a37854f069d0a3911b5c6550b67c0eb9b95aec72db57b4e6fdcab425fa0b44dd
Jul 28 21:34:51.468149 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:51.468112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds" event={"ID":"904d0185-10fc-4a44-a66c-7bfd6df09936","Type":"ContainerStarted","Data":"e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d"}
Jul 28 21:34:51.468296 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:51.468158    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds" event={"ID":"904d0185-10fc-4a44-a66c-7bfd6df09936","Type":"ContainerStarted","Data":"a37854f069d0a3911b5c6550b67c0eb9b95aec72db57b4e6fdcab425fa0b44dd"}
Jul 28 21:34:51.487714 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:34:51.487661    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds" podStartSLOduration=1.487640737 podStartE2EDuration="1.487640737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:34:51.486349434 +0000 UTC m=+2980.341193346" watchObservedRunningTime="2026-07-28 21:34:51.487640737 +0000 UTC m=+2980.342484648"
Jul 28 21:35:00.579236 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:35:00.579193    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:35:00.579752 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:35:00.579275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:35:00.592049 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:35:00.592008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:35:01.533563 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:35:01.533533    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:36:39.514897 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:39.514806    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"]
Jul 28 21:36:54.536886 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.536840    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds" podUID="904d0185-10fc-4a44-a66c-7bfd6df09936" containerName="main" containerID="cri-o://e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d" gracePeriod=45
Jul 28 21:36:54.796745 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.796677    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:36:54.871732 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.871700    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-dshm\") pod \"904d0185-10fc-4a44-a66c-7bfd6df09936\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") "
Jul 28 21:36:54.871911 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.871770    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904d0185-10fc-4a44-a66c-7bfd6df09936-tls-certs\") pod \"904d0185-10fc-4a44-a66c-7bfd6df09936\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") "
Jul 28 21:36:54.871911 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.871792    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-tmp-dir\") pod \"904d0185-10fc-4a44-a66c-7bfd6df09936\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") "
Jul 28 21:36:54.871911 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.871806    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-home\") pod \"904d0185-10fc-4a44-a66c-7bfd6df09936\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") "
Jul 28 21:36:54.871911 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.871822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-model-cache\") pod \"904d0185-10fc-4a44-a66c-7bfd6df09936\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") "
Jul 28 21:36:54.871911 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.871856    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n454d\" (UniqueName: \"kubernetes.io/projected/904d0185-10fc-4a44-a66c-7bfd6df09936-kube-api-access-n454d\") pod \"904d0185-10fc-4a44-a66c-7bfd6df09936\" (UID: \"904d0185-10fc-4a44-a66c-7bfd6df09936\") "
Jul 28 21:36:54.872221 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.872038    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "904d0185-10fc-4a44-a66c-7bfd6df09936" (UID: "904d0185-10fc-4a44-a66c-7bfd6df09936"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:36:54.872221 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.872062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-home" (OuterVolumeSpecName: "home") pod "904d0185-10fc-4a44-a66c-7bfd6df09936" (UID: "904d0185-10fc-4a44-a66c-7bfd6df09936"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:36:54.872221 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.872177    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-model-cache" (OuterVolumeSpecName: "model-cache") pod "904d0185-10fc-4a44-a66c-7bfd6df09936" (UID: "904d0185-10fc-4a44-a66c-7bfd6df09936"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:36:54.872221 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.872212    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-tmp-dir\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:36:54.872357 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.872227    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-home\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:36:54.874093 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.874056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-dshm" (OuterVolumeSpecName: "dshm") pod "904d0185-10fc-4a44-a66c-7bfd6df09936" (UID: "904d0185-10fc-4a44-a66c-7bfd6df09936"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:36:54.874093 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.874065    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904d0185-10fc-4a44-a66c-7bfd6df09936-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "904d0185-10fc-4a44-a66c-7bfd6df09936" (UID: "904d0185-10fc-4a44-a66c-7bfd6df09936"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:36:54.874259 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.874158    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904d0185-10fc-4a44-a66c-7bfd6df09936-kube-api-access-n454d" (OuterVolumeSpecName: "kube-api-access-n454d") pod "904d0185-10fc-4a44-a66c-7bfd6df09936" (UID: "904d0185-10fc-4a44-a66c-7bfd6df09936"). InnerVolumeSpecName "kube-api-access-n454d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:36:54.973574 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.973542    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-dshm\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:36:54.973574 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.973571    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904d0185-10fc-4a44-a66c-7bfd6df09936-tls-certs\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:36:54.973574 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.973581    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904d0185-10fc-4a44-a66c-7bfd6df09936-model-cache\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:36:54.973810 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:54.973590    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n454d\" (UniqueName: \"kubernetes.io/projected/904d0185-10fc-4a44-a66c-7bfd6df09936-kube-api-access-n454d\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:36:55.083930 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:55.083841    2578 generic.go:356] "Generic (PLEG): container finished" podID="904d0185-10fc-4a44-a66c-7bfd6df09936" containerID="e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d" exitCode=0
Jul 28 21:36:55.083930 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:55.083898    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"
Jul 28 21:36:55.084165 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:55.083937    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds" event={"ID":"904d0185-10fc-4a44-a66c-7bfd6df09936","Type":"ContainerDied","Data":"e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d"}
Jul 28 21:36:55.084165 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:55.083992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds" event={"ID":"904d0185-10fc-4a44-a66c-7bfd6df09936","Type":"ContainerDied","Data":"a37854f069d0a3911b5c6550b67c0eb9b95aec72db57b4e6fdcab425fa0b44dd"}
Jul 28 21:36:55.084165 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:55.084018    2578 scope.go:117] "RemoveContainer" containerID="e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d"
Jul 28 21:36:55.093070 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:55.093052    2578 scope.go:117] "RemoveContainer" containerID="e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d"
Jul 28 21:36:55.093412 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:36:55.093388    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d\": container with ID starting with e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d not found: ID does not exist" containerID="e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d"
Jul 28 21:36:55.093478 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:55.093422    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d"} err="failed to get container status \"e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d\": rpc error: code = NotFound desc = could not find container \"e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d\": container with ID starting with e78cd4e69e52f18b4737517f6212c35112eef70e8b1b7b8b303d8b3903dedb7d not found: ID does not exist"
Jul 28 21:36:55.108112 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:55.108057    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"]
Jul 28 21:36:55.112090 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:55.112051    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b439d9/rate-test-model-b439d9-kserve-86d9978fb7-pltds"]
Jul 28 21:36:55.529837 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:36:55.529806    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="904d0185-10fc-4a44-a66c-7bfd6df09936" path="/var/lib/kubelet/pods/904d0185-10fc-4a44-a66c-7bfd6df09936/volumes"
Jul 28 21:37:29.258042 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.258004    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"]
Jul 28 21:37:29.258485 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.258412    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904d0185-10fc-4a44-a66c-7bfd6df09936" containerName="main"
Jul 28 21:37:29.258485 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.258425    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="904d0185-10fc-4a44-a66c-7bfd6df09936" containerName="main"
Jul 28 21:37:29.258572 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.258502    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="904d0185-10fc-4a44-a66c-7bfd6df09936" containerName="main"
Jul 28 21:37:29.261690 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.261656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.263968 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.263947    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-35d85b-openshift-default-dockercfg-gdnn9\""
Jul 28 21:37:29.275694 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.275668    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"]
Jul 28 21:37:29.296481 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.296452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istiod-ca-cert\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.296604 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.296521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-certs\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.296604 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.296590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-token\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.296700 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.296634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-envoy\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.296741 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.296701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-data\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.296788 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.296740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cv8h\" (UniqueName: \"kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-kube-api-access-5cv8h\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.296788 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.296769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-socket\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.296870 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.296794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-credential-socket\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.296870 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.296824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-podinfo\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.397403 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.397363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-certs\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.397403 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.397408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-token\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.397663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.397425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-envoy\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.397663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.397469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-data\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.397663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.397493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cv8h\" (UniqueName: \"kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-kube-api-access-5cv8h\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.397663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.397508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-socket\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.397663 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.397529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-credential-socket\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.397941 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.397749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-podinfo\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.398051 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.398027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-credential-socket\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.398139 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.397849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-certs\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.398139 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.398071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-data\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.398139 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.397922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-socket\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.398139 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.398119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istiod-ca-cert\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.398657 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.398636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istiod-ca-cert\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.400238 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.400215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-envoy\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.400424 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.400398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-podinfo\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.405371 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.405344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cv8h\" (UniqueName: \"kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-kube-api-access-5cv8h\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.405489 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.405391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-token\") pod \"e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") " pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.571893 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.571812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:29.710625 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.710599    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"]
Jul 28 21:37:29.713226 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:37:29.713192    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7478ddd5_4e4d_4b5c_97cc_0cc339e1fa9e.slice/crio-7896f3a64a6a95489f86620cc551dbf160cfb3d4db466ac8c82b5412a04da5d5 WatchSource:0}: Error finding container 7896f3a64a6a95489f86620cc551dbf160cfb3d4db466ac8c82b5412a04da5d5: Status 404 returned error can't find the container with id 7896f3a64a6a95489f86620cc551dbf160cfb3d4db466ac8c82b5412a04da5d5
Jul 28 21:37:29.716000 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.715968    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 21:37:29.716100 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.716035    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 21:37:29.716100 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:29.716064    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 21:37:30.261673 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:30.261640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" event={"ID":"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e","Type":"ContainerStarted","Data":"9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83"}
Jul 28 21:37:30.262192 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:30.261689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" event={"ID":"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e","Type":"ContainerStarted","Data":"7896f3a64a6a95489f86620cc551dbf160cfb3d4db466ac8c82b5412a04da5d5"}
Jul 28 21:37:30.285541 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:30.285484    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" podStartSLOduration=1.285464249 podStartE2EDuration="1.285464249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:37:30.283540795 +0000 UTC m=+3139.138384735" watchObservedRunningTime="2026-07-28 21:37:30.285464249 +0000 UTC m=+3139.140308162"
Jul 28 21:37:30.573056 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:30.572959    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:30.574553 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:30.574526    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 28 21:37:30.574677 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:30.574586    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" podUID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 28 21:37:31.577754 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:31.577724    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:32.271608 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:32.271576    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:32.272567 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:32.272550    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:34.190538 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.190496    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"]
Jul 28 21:37:34.194264 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.194243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:37:34.196664 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.196640    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-35d85b\""
Jul 28 21:37:34.196664 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.196649    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-35d85b-dockercfg-d6jrq\""
Jul 28 21:37:34.204202 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.204179    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"]
Jul 28 21:37:34.246910 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.246874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf\" (UID: \"2fb3b398-adca-4d22-8777-d1d02f7d0bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:37:34.247038 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.247020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2cbr\" (UniqueName: \"kubernetes.io/projected/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-kube-api-access-g2cbr\") pod \"payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf\" (UID: \"2fb3b398-adca-4d22-8777-d1d02f7d0bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:37:34.348033 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.347994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cbr\" (UniqueName: \"kubernetes.io/projected/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-kube-api-access-g2cbr\") pod \"payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf\" (UID: \"2fb3b398-adca-4d22-8777-d1d02f7d0bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:37:34.348267 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.348201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf\" (UID: \"2fb3b398-adca-4d22-8777-d1d02f7d0bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:37:34.348895 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.348877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf\" (UID: \"2fb3b398-adca-4d22-8777-d1d02f7d0bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:37:34.356648 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.356622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cbr\" (UniqueName: \"kubernetes.io/projected/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-kube-api-access-g2cbr\") pod \"payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf\" (UID: \"2fb3b398-adca-4d22-8777-d1d02f7d0bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:37:34.506869 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.506835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:37:34.645325 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:34.645297    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"]
Jul 28 21:37:34.648267 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:37:34.648178    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb3b398_adca_4d22_8777_d1d02f7d0bd2.slice/crio-cd5828013ab25e6051f70322e1b5bf7a200f6dafa8a03bda434640bd8114ac3d WatchSource:0}: Error finding container cd5828013ab25e6051f70322e1b5bf7a200f6dafa8a03bda434640bd8114ac3d: Status 404 returned error can't find the container with id cd5828013ab25e6051f70322e1b5bf7a200f6dafa8a03bda434640bd8114ac3d
Jul 28 21:37:35.289833 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:35.289798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf" event={"ID":"2fb3b398-adca-4d22-8777-d1d02f7d0bd2","Type":"ContainerStarted","Data":"ec6ec6cb30dbb76c7cca34a54a99b70c4a7add5d2132ee25a580b2391efbd9d6"}
Jul 28 21:37:35.290298 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:35.289859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf" event={"ID":"2fb3b398-adca-4d22-8777-d1d02f7d0bd2","Type":"ContainerStarted","Data":"cd5828013ab25e6051f70322e1b5bf7a200f6dafa8a03bda434640bd8114ac3d"}
Jul 28 21:37:35.290298 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:35.289897    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:37:35.305730 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:35.305685    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf" podStartSLOduration=1.305670267 podStartE2EDuration="1.305670267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:37:35.304930356 +0000 UTC m=+3144.159774268" watchObservedRunningTime="2026-07-28 21:37:35.305670267 +0000 UTC m=+3144.160514178"
Jul 28 21:37:46.297741 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:46.297696    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:37:50.367942 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:50.367852    2578 generic.go:356] "Generic (PLEG): container finished" podID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerID="9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83" exitCode=0
Jul 28 21:37:50.367942 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:50.367926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" event={"ID":"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e","Type":"ContainerDied","Data":"9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83"}
Jul 28 21:37:50.368425 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:50.368367    2578 scope.go:117] "RemoveContainer" containerID="9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83"
Jul 28 21:37:50.369635 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:50.369597    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 21:37:50.369716 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:50.369700    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 21:37:50.369761 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:50.369750    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 21:37:51.374647 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:51.374604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" event={"ID":"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e","Type":"ContainerStarted","Data":"18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff"}
Jul 28 21:37:51.572266 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:51.572236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:51.572559 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:51.572534    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 28 21:37:51.572656 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:51.572599    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" podUID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 28 21:37:52.572442 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:52.572406    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 28 21:37:52.572843 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:52.572460    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" podUID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 28 21:37:53.572530 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:53.572500    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 28 21:37:53.572918 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:53.572551    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" podUID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 28 21:37:54.151904 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:54.151854    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-76zbd_73fe5e12-e5b9-4359-8a5b-342ff454649e/payload-processing/0.log"
Jul 28 21:37:54.572851 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:54.572817    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 28 21:37:54.573262 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:54.572901    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" podUID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 28 21:37:55.576749 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:55.576718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:55.577220 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:55.577113    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:37:55.578005 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:37:55.577986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:38:00.339163 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.339128    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"]
Jul 28 21:38:00.343483 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.343460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.346260 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.346225    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7aec84\"/\"openshift-service-ca.crt\""
Jul 28 21:38:00.346438 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.346278    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7aec84\"/\"kube-root-ca.crt\""
Jul 28 21:38:00.346438 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.346409    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7aec84\"/\"ipp-route-7aec84-kserve-self-signed-certs\""
Jul 28 21:38:00.347165 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.347149    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7aec84\"/\"default-dockercfg-sxp74\""
Jul 28 21:38:00.356034 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.356011    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"]
Jul 28 21:38:00.400468 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.400427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tmp-dir\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.400652 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.400475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-home\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.400652 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.400501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ldmw\" (UniqueName: \"kubernetes.io/projected/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-kube-api-access-7ldmw\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.400652 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.400603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-model-cache\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.400796 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.400670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tls-certs\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.400796 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.400715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-dshm\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.501912 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.501876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-model-cache\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.502108 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.501931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tls-certs\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.502108 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.501990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-dshm\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.502108 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.502029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tmp-dir\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.502108 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.502064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-home\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.502336 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.502110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ldmw\" (UniqueName: \"kubernetes.io/projected/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-kube-api-access-7ldmw\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.502395 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.502356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-model-cache\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.502493 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.502477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tmp-dir\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.502591 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.502570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-home\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.504376 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.504354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-dshm\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.504629 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.504612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tls-certs\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.510634 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.510612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ldmw\" (UniqueName: \"kubernetes.io/projected/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-kube-api-access-7ldmw\") pod \"ipp-route-7aec84-kserve-675f48ffff-4g5s5\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") " pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.652995 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.652900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:00.808728 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:00.808697    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"]
Jul 28 21:38:00.810374 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:38:00.810343    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f0f063a_2325_4bae_a3c7_d320ed50c4e2.slice/crio-58a1627a623b2688643defad83e9922afda35a48fee9d6a2286441e049318684 WatchSource:0}: Error finding container 58a1627a623b2688643defad83e9922afda35a48fee9d6a2286441e049318684: Status 404 returned error can't find the container with id 58a1627a623b2688643defad83e9922afda35a48fee9d6a2286441e049318684
Jul 28 21:38:01.433440 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:01.433397    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5" event={"ID":"6f0f063a-2325-4bae-a3c7-d320ed50c4e2","Type":"ContainerStarted","Data":"8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74"}
Jul 28 21:38:01.433840 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:01.433451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5" event={"ID":"6f0f063a-2325-4bae-a3c7-d320ed50c4e2","Type":"ContainerStarted","Data":"58a1627a623b2688643defad83e9922afda35a48fee9d6a2286441e049318684"}
Jul 28 21:38:01.455540 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:01.455496    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5" podStartSLOduration=1.455481499 podStartE2EDuration="1.455481499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:01.454719934 +0000 UTC m=+3170.309563846" watchObservedRunningTime="2026-07-28 21:38:01.455481499 +0000 UTC m=+3170.310325409"
Jul 28 21:38:10.653467 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:10.653435    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:10.653880 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:10.653484    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:10.666872 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:10.666839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:11.501955 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:11.501925    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:38:46.579275 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:46.579244    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-76zbd_73fe5e12-e5b9-4359-8a5b-342ff454649e/payload-processing/0.log"
Jul 28 21:38:52.032620 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.032574    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"]
Jul 28 21:38:52.036596 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.036574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:38:52.039236 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.039216    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d4bcd2-dockercfg-9l9f2\""
Jul 28 21:38:52.039236 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.039225    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d4bcd2\""
Jul 28 21:38:52.049256 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.049235    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"]
Jul 28 21:38:52.052727 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.052705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"]
Jul 28 21:38:52.052839 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.052783    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:38:52.065156 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.065134    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"]
Jul 28 21:38:52.092412 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.092388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w954q\" (UniqueName: \"kubernetes.io/projected/9f90eec3-08ea-44b1-b281-fce597418776-kube-api-access-w954q\") pod \"payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm\" (UID: \"9f90eec3-08ea-44b1-b281-fce597418776\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:38:52.092535 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.092494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f90eec3-08ea-44b1-b281-fce597418776-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm\" (UID: \"9f90eec3-08ea-44b1-b281-fce597418776\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:38:52.193384 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.193345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w954q\" (UniqueName: \"kubernetes.io/projected/9f90eec3-08ea-44b1-b281-fce597418776-kube-api-access-w954q\") pod \"payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm\" (UID: \"9f90eec3-08ea-44b1-b281-fce597418776\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:38:52.193558 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.193482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f90eec3-08ea-44b1-b281-fce597418776-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm\" (UID: \"9f90eec3-08ea-44b1-b281-fce597418776\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:38:52.193558 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.193525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45559\" (UniqueName: \"kubernetes.io/projected/ee98977a-0955-422b-8f19-ffea573d3b90-kube-api-access-45559\") pod \"payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv\" (UID: \"ee98977a-0955-422b-8f19-ffea573d3b90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:38:52.193633 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.193569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee98977a-0955-422b-8f19-ffea573d3b90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv\" (UID: \"ee98977a-0955-422b-8f19-ffea573d3b90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:38:52.194291 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.194270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f90eec3-08ea-44b1-b281-fce597418776-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm\" (UID: \"9f90eec3-08ea-44b1-b281-fce597418776\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:38:52.202096 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.202060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w954q\" (UniqueName: \"kubernetes.io/projected/9f90eec3-08ea-44b1-b281-fce597418776-kube-api-access-w954q\") pod \"payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm\" (UID: \"9f90eec3-08ea-44b1-b281-fce597418776\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:38:52.294931 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.294847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45559\" (UniqueName: \"kubernetes.io/projected/ee98977a-0955-422b-8f19-ffea573d3b90-kube-api-access-45559\") pod \"payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv\" (UID: \"ee98977a-0955-422b-8f19-ffea573d3b90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:38:52.294931 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.294902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee98977a-0955-422b-8f19-ffea573d3b90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv\" (UID: \"ee98977a-0955-422b-8f19-ffea573d3b90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:38:52.295595 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.295575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee98977a-0955-422b-8f19-ffea573d3b90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv\" (UID: \"ee98977a-0955-422b-8f19-ffea573d3b90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:38:52.303272 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.303250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45559\" (UniqueName: \"kubernetes.io/projected/ee98977a-0955-422b-8f19-ffea573d3b90-kube-api-access-45559\") pod \"payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv\" (UID: \"ee98977a-0955-422b-8f19-ffea573d3b90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:38:52.347482 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.347457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:38:52.364566 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.364533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:38:52.505731 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.505705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"]
Jul 28 21:38:52.508655 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:38:52.508629    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f90eec3_08ea_44b1_b281_fce597418776.slice/crio-068963905f0343c4ced0c1c7f410de01f65e5e958887f2f14f430f060eca7e59 WatchSource:0}: Error finding container 068963905f0343c4ced0c1c7f410de01f65e5e958887f2f14f430f060eca7e59: Status 404 returned error can't find the container with id 068963905f0343c4ced0c1c7f410de01f65e5e958887f2f14f430f060eca7e59
Jul 28 21:38:52.534899 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.534863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"]
Jul 28 21:38:52.542205 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:38:52.542181    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee98977a_0955_422b_8f19_ffea573d3b90.slice/crio-9e9672bb8785ce9b215e388e1a75a5b8e6d3b7f72a0e4f8dd2f9f1d8a366e6dd WatchSource:0}: Error finding container 9e9672bb8785ce9b215e388e1a75a5b8e6d3b7f72a0e4f8dd2f9f1d8a366e6dd: Status 404 returned error can't find the container with id 9e9672bb8785ce9b215e388e1a75a5b8e6d3b7f72a0e4f8dd2f9f1d8a366e6dd
Jul 28 21:38:52.700814 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.700761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv" event={"ID":"ee98977a-0955-422b-8f19-ffea573d3b90","Type":"ContainerStarted","Data":"1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338"}
Jul 28 21:38:52.701013 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.700876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:38:52.701013 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.700901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv" event={"ID":"ee98977a-0955-422b-8f19-ffea573d3b90","Type":"ContainerStarted","Data":"9e9672bb8785ce9b215e388e1a75a5b8e6d3b7f72a0e4f8dd2f9f1d8a366e6dd"}
Jul 28 21:38:52.702519 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.702485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm" event={"ID":"9f90eec3-08ea-44b1-b281-fce597418776","Type":"ContainerStarted","Data":"28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8"}
Jul 28 21:38:52.702647 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.702529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm" event={"ID":"9f90eec3-08ea-44b1-b281-fce597418776","Type":"ContainerStarted","Data":"068963905f0343c4ced0c1c7f410de01f65e5e958887f2f14f430f060eca7e59"}
Jul 28 21:38:52.702647 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.702575    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:38:52.718703 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.718636    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv" podStartSLOduration=0.718621298 podStartE2EDuration="718.621298ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:52.717249916 +0000 UTC m=+3221.572093850" watchObservedRunningTime="2026-07-28 21:38:52.718621298 +0000 UTC m=+3221.573465209"
Jul 28 21:38:52.733558 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:38:52.733505    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm" podStartSLOduration=0.733483821 podStartE2EDuration="733.483821ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:52.733154857 +0000 UTC m=+3221.587998769" watchObservedRunningTime="2026-07-28 21:38:52.733483821 +0000 UTC m=+3221.588327733"
Jul 28 21:39:03.710487 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:03.710391    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm" podUID="9f90eec3-08ea-44b1-b281-fce597418776" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.61:9004: connect: connection refused"
Jul 28 21:39:03.710963 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:03.710828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:39:13.709114 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:13.709064    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:39:21.747263 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:21.747222    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"]
Jul 28 21:39:21.747789 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:21.747499    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv" podUID="ee98977a-0955-422b-8f19-ffea573d3b90" containerName="payload-processing" containerID="cri-o://1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338" gracePeriod=30
Jul 28 21:39:21.753915 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:21.753886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"]
Jul 28 21:39:21.754235 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:21.754124    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm" podUID="9f90eec3-08ea-44b1-b281-fce597418776" containerName="payload-pre-processing" containerID="cri-o://28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8" gracePeriod=30
Jul 28 21:39:22.019021 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.018989    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:39:22.044803 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.044772    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:39:22.084443 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.084408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee98977a-0955-422b-8f19-ffea573d3b90-plugins-config-volume\") pod \"ee98977a-0955-422b-8f19-ffea573d3b90\" (UID: \"ee98977a-0955-422b-8f19-ffea573d3b90\") "
Jul 28 21:39:22.084616 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.084470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w954q\" (UniqueName: \"kubernetes.io/projected/9f90eec3-08ea-44b1-b281-fce597418776-kube-api-access-w954q\") pod \"9f90eec3-08ea-44b1-b281-fce597418776\" (UID: \"9f90eec3-08ea-44b1-b281-fce597418776\") "
Jul 28 21:39:22.084616 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.084505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f90eec3-08ea-44b1-b281-fce597418776-plugins-config-volume\") pod \"9f90eec3-08ea-44b1-b281-fce597418776\" (UID: \"9f90eec3-08ea-44b1-b281-fce597418776\") "
Jul 28 21:39:22.084616 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.084548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45559\" (UniqueName: \"kubernetes.io/projected/ee98977a-0955-422b-8f19-ffea573d3b90-kube-api-access-45559\") pod \"ee98977a-0955-422b-8f19-ffea573d3b90\" (UID: \"ee98977a-0955-422b-8f19-ffea573d3b90\") "
Jul 28 21:39:22.084888 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.084862    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee98977a-0955-422b-8f19-ffea573d3b90-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee98977a-0955-422b-8f19-ffea573d3b90" (UID: "ee98977a-0955-422b-8f19-ffea573d3b90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:39:22.084985 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.084916    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f90eec3-08ea-44b1-b281-fce597418776-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f90eec3-08ea-44b1-b281-fce597418776" (UID: "9f90eec3-08ea-44b1-b281-fce597418776"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:39:22.085039 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.085021    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee98977a-0955-422b-8f19-ffea573d3b90-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:39:22.085113 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.085040    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f90eec3-08ea-44b1-b281-fce597418776-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:39:22.086901 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.086852    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee98977a-0955-422b-8f19-ffea573d3b90-kube-api-access-45559" (OuterVolumeSpecName: "kube-api-access-45559") pod "ee98977a-0955-422b-8f19-ffea573d3b90" (UID: "ee98977a-0955-422b-8f19-ffea573d3b90"). InnerVolumeSpecName "kube-api-access-45559". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:39:22.087015 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.086956    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f90eec3-08ea-44b1-b281-fce597418776-kube-api-access-w954q" (OuterVolumeSpecName: "kube-api-access-w954q") pod "9f90eec3-08ea-44b1-b281-fce597418776" (UID: "9f90eec3-08ea-44b1-b281-fce597418776"). InnerVolumeSpecName "kube-api-access-w954q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:39:22.186136 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.186101    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w954q\" (UniqueName: \"kubernetes.io/projected/9f90eec3-08ea-44b1-b281-fce597418776-kube-api-access-w954q\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:39:22.186136 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.186135    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45559\" (UniqueName: \"kubernetes.io/projected/ee98977a-0955-422b-8f19-ffea573d3b90-kube-api-access-45559\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:39:22.857850 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.857811    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee98977a-0955-422b-8f19-ffea573d3b90" containerID="1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338" exitCode=0
Jul 28 21:39:22.858367 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.857855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv" event={"ID":"ee98977a-0955-422b-8f19-ffea573d3b90","Type":"ContainerDied","Data":"1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338"}
Jul 28 21:39:22.858367 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.857924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv" event={"ID":"ee98977a-0955-422b-8f19-ffea573d3b90","Type":"ContainerDied","Data":"9e9672bb8785ce9b215e388e1a75a5b8e6d3b7f72a0e4f8dd2f9f1d8a366e6dd"}
Jul 28 21:39:22.858367 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.857864    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"
Jul 28 21:39:22.858367 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.857956    2578 scope.go:117] "RemoveContainer" containerID="1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338"
Jul 28 21:39:22.859463 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.859404    2578 generic.go:356] "Generic (PLEG): container finished" podID="9f90eec3-08ea-44b1-b281-fce597418776" containerID="28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8" exitCode=0
Jul 28 21:39:22.859463 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.859435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm" event={"ID":"9f90eec3-08ea-44b1-b281-fce597418776","Type":"ContainerDied","Data":"28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8"}
Jul 28 21:39:22.859463 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.859458    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"
Jul 28 21:39:22.859607 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.859480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm" event={"ID":"9f90eec3-08ea-44b1-b281-fce597418776","Type":"ContainerDied","Data":"068963905f0343c4ced0c1c7f410de01f65e5e958887f2f14f430f060eca7e59"}
Jul 28 21:39:22.868225 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.868205    2578 scope.go:117] "RemoveContainer" containerID="1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338"
Jul 28 21:39:22.868505 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:39:22.868478    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338\": container with ID starting with 1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338 not found: ID does not exist" containerID="1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338"
Jul 28 21:39:22.868624 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.868516    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338"} err="failed to get container status \"1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338\": rpc error: code = NotFound desc = could not find container \"1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338\": container with ID starting with 1be6be01fb5b35a16b30fc7f9c01c54e3485e39710841c3bc9393df0e7dcd338 not found: ID does not exist"
Jul 28 21:39:22.868624 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.868534    2578 scope.go:117] "RemoveContainer" containerID="28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8"
Jul 28 21:39:22.876899 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.876883    2578 scope.go:117] "RemoveContainer" containerID="28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8"
Jul 28 21:39:22.877193 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:39:22.877175    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8\": container with ID starting with 28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8 not found: ID does not exist" containerID="28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8"
Jul 28 21:39:22.877243 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.877199    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8"} err="failed to get container status \"28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8\": rpc error: code = NotFound desc = could not find container \"28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8\": container with ID starting with 28e0ce28992817496637418b80d531691219a4f9f711ca869c300bc19a5dcec8 not found: ID does not exist"
Jul 28 21:39:22.886310 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.886288    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"]
Jul 28 21:39:22.891529 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.891507    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d4bcd2-7c7765d47b-vb9qm"]
Jul 28 21:39:22.901766 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.901738    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"]
Jul 28 21:39:22.903960 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:22.903942    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d4bcd2-694475d9b6-spntv"]
Jul 28 21:39:23.530066 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:23.530035    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f90eec3-08ea-44b1-b281-fce597418776" path="/var/lib/kubelet/pods/9f90eec3-08ea-44b1-b281-fce597418776/volumes"
Jul 28 21:39:23.530456 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:23.530439    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee98977a-0955-422b-8f19-ffea573d3b90" path="/var/lib/kubelet/pods/ee98977a-0955-422b-8f19-ffea573d3b90/volumes"
Jul 28 21:39:50.420815 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:39:50.420775    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"]
Jul 28 21:40:05.440700 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.440636    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5" podUID="6f0f063a-2325-4bae-a3c7-d320ed50c4e2" containerName="main" containerID="cri-o://8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74" gracePeriod=45
Jul 28 21:40:05.704472 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.704407    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:40:05.816036 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-model-cache\") pod \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") "
Jul 28 21:40:05.816286 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816108    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-home\") pod \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") "
Jul 28 21:40:05.816286 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ldmw\" (UniqueName: \"kubernetes.io/projected/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-kube-api-access-7ldmw\") pod \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") "
Jul 28 21:40:05.816286 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816185    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tmp-dir\") pod \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") "
Jul 28 21:40:05.816286 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816204    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-dshm\") pod \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") "
Jul 28 21:40:05.816286 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816248    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tls-certs\") pod \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\" (UID: \"6f0f063a-2325-4bae-a3c7-d320ed50c4e2\") "
Jul 28 21:40:05.816548 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816330    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f0f063a-2325-4bae-a3c7-d320ed50c4e2" (UID: "6f0f063a-2325-4bae-a3c7-d320ed50c4e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:40:05.816548 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816414    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-home" (OuterVolumeSpecName: "home") pod "6f0f063a-2325-4bae-a3c7-d320ed50c4e2" (UID: "6f0f063a-2325-4bae-a3c7-d320ed50c4e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:40:05.816548 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816443    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f0f063a-2325-4bae-a3c7-d320ed50c4e2" (UID: "6f0f063a-2325-4bae-a3c7-d320ed50c4e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:40:05.816709 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816591    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-model-cache\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:40:05.816709 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816604    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-home\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:40:05.816709 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.816612    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tmp-dir\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:40:05.818662 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.818640    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-dshm" (OuterVolumeSpecName: "dshm") pod "6f0f063a-2325-4bae-a3c7-d320ed50c4e2" (UID: "6f0f063a-2325-4bae-a3c7-d320ed50c4e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:40:05.818735 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.818676    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f0f063a-2325-4bae-a3c7-d320ed50c4e2" (UID: "6f0f063a-2325-4bae-a3c7-d320ed50c4e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:40:05.818776 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.818747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-kube-api-access-7ldmw" (OuterVolumeSpecName: "kube-api-access-7ldmw") pod "6f0f063a-2325-4bae-a3c7-d320ed50c4e2" (UID: "6f0f063a-2325-4bae-a3c7-d320ed50c4e2"). InnerVolumeSpecName "kube-api-access-7ldmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:40:05.917257 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.917213    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ldmw\" (UniqueName: \"kubernetes.io/projected/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-kube-api-access-7ldmw\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:40:05.917257 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.917249    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-dshm\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:40:05.917257 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:05.917260    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0f063a-2325-4bae-a3c7-d320ed50c4e2-tls-certs\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:40:06.082817 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:06.082780    2578 generic.go:356] "Generic (PLEG): container finished" podID="6f0f063a-2325-4bae-a3c7-d320ed50c4e2" containerID="8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74" exitCode=0
Jul 28 21:40:06.082817 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:06.082819    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"
Jul 28 21:40:06.083128 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:06.082842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5" event={"ID":"6f0f063a-2325-4bae-a3c7-d320ed50c4e2","Type":"ContainerDied","Data":"8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74"}
Jul 28 21:40:06.083128 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:06.082888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5" event={"ID":"6f0f063a-2325-4bae-a3c7-d320ed50c4e2","Type":"ContainerDied","Data":"58a1627a623b2688643defad83e9922afda35a48fee9d6a2286441e049318684"}
Jul 28 21:40:06.083128 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:06.082911    2578 scope.go:117] "RemoveContainer" containerID="8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74"
Jul 28 21:40:06.092427 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:06.092409    2578 scope.go:117] "RemoveContainer" containerID="8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74"
Jul 28 21:40:06.092668 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:40:06.092648    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74\": container with ID starting with 8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74 not found: ID does not exist" containerID="8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74"
Jul 28 21:40:06.092734 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:06.092678    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74"} err="failed to get container status \"8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74\": rpc error: code = NotFound desc = could not find container \"8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74\": container with ID starting with 8645023b8905be913a4b71a86477116a292767834c47632c2cf00c84e4f8ea74 not found: ID does not exist"
Jul 28 21:40:06.104173 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:06.104148    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"]
Jul 28 21:40:06.107791 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:06.107767    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7aec84/ipp-route-7aec84-kserve-675f48ffff-4g5s5"]
Jul 28 21:40:07.529699 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:07.529666    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f0f063a-2325-4bae-a3c7-d320ed50c4e2" path="/var/lib/kubelet/pods/6f0f063a-2325-4bae-a3c7-d320ed50c4e2/volumes"
Jul 28 21:40:57.106030 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.105993    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"]
Jul 28 21:40:57.106603 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.106261    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf" podUID="2fb3b398-adca-4d22-8777-d1d02f7d0bd2" containerName="payload-pre-processing" containerID="cri-o://ec6ec6cb30dbb76c7cca34a54a99b70c4a7add5d2132ee25a580b2391efbd9d6" gracePeriod=30
Jul 28 21:40:57.342376 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.342344    2578 generic.go:356] "Generic (PLEG): container finished" podID="2fb3b398-adca-4d22-8777-d1d02f7d0bd2" containerID="ec6ec6cb30dbb76c7cca34a54a99b70c4a7add5d2132ee25a580b2391efbd9d6" exitCode=0
Jul 28 21:40:57.342548 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.342421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf" event={"ID":"2fb3b398-adca-4d22-8777-d1d02f7d0bd2","Type":"ContainerDied","Data":"ec6ec6cb30dbb76c7cca34a54a99b70c4a7add5d2132ee25a580b2391efbd9d6"}
Jul 28 21:40:57.380061 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.380037    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:40:57.499550 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.499518    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-plugins-config-volume\") pod \"2fb3b398-adca-4d22-8777-d1d02f7d0bd2\" (UID: \"2fb3b398-adca-4d22-8777-d1d02f7d0bd2\") "
Jul 28 21:40:57.499730 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.499569    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2cbr\" (UniqueName: \"kubernetes.io/projected/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-kube-api-access-g2cbr\") pod \"2fb3b398-adca-4d22-8777-d1d02f7d0bd2\" (UID: \"2fb3b398-adca-4d22-8777-d1d02f7d0bd2\") "
Jul 28 21:40:57.500002 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.499975    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2fb3b398-adca-4d22-8777-d1d02f7d0bd2" (UID: "2fb3b398-adca-4d22-8777-d1d02f7d0bd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:40:57.501886 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.501866    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-kube-api-access-g2cbr" (OuterVolumeSpecName: "kube-api-access-g2cbr") pod "2fb3b398-adca-4d22-8777-d1d02f7d0bd2" (UID: "2fb3b398-adca-4d22-8777-d1d02f7d0bd2"). InnerVolumeSpecName "kube-api-access-g2cbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:40:57.601033 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.601000    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:40:57.601033 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:57.601027    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2cbr\" (UniqueName: \"kubernetes.io/projected/2fb3b398-adca-4d22-8777-d1d02f7d0bd2-kube-api-access-g2cbr\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:40:58.356742 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:58.356715    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"
Jul 28 21:40:58.357252 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:58.356711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf" event={"ID":"2fb3b398-adca-4d22-8777-d1d02f7d0bd2","Type":"ContainerDied","Data":"cd5828013ab25e6051f70322e1b5bf7a200f6dafa8a03bda434640bd8114ac3d"}
Jul 28 21:40:58.357252 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:58.356952    2578 scope.go:117] "RemoveContainer" containerID="ec6ec6cb30dbb76c7cca34a54a99b70c4a7add5d2132ee25a580b2391efbd9d6"
Jul 28 21:40:58.375280 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:58.375229    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"]
Jul 28 21:40:58.377693 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:58.377668    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-35d85b-6ff97f5fb-kf4zf"]
Jul 28 21:40:59.530335 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:40:59.530303    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fb3b398-adca-4d22-8777-d1d02f7d0bd2" path="/var/lib/kubelet/pods/2fb3b398-adca-4d22-8777-d1d02f7d0bd2/volumes"
Jul 28 21:41:04.317495 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:04.317462    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"]
Jul 28 21:41:04.317930 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:04.317707    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" podUID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerName="istio-proxy" containerID="cri-o://18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff" gracePeriod=30
Jul 28 21:41:09.581493 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.581426    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:41:09.722053 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-certs\") pod \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") "
Jul 28 21:41:09.722053 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722054    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-data\") pod \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") "
Jul 28 21:41:09.722297 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722071    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-credential-socket\") pod \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") "
Jul 28 21:41:09.722297 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722121    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cv8h\" (UniqueName: \"kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-kube-api-access-5cv8h\") pod \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") "
Jul 28 21:41:09.722297 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-token\") pod \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") "
Jul 28 21:41:09.722297 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722228    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-podinfo\") pod \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") "
Jul 28 21:41:09.722297 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722293    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-socket\") pod \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") "
Jul 28 21:41:09.722569 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722320    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" (UID: "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:41:09.722569 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722359    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-envoy\") pod \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") "
Jul 28 21:41:09.722569 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" (UID: "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:41:09.722569 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722417    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istiod-ca-cert\") pod \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\" (UID: \"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e\") "
Jul 28 21:41:09.722799 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722600    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" (UID: "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:41:09.722799 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722752    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-socket\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:09.722799 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722772    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-workload-certs\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:09.722799 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722786    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-credential-socket\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:09.723164 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.722891    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" (UID: "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:41:09.723164 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.723164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" (UID: "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:41:09.724515 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.724488    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" (UID: "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:41:09.724637 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.724539    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" (UID: "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 21:41:09.724637 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.724553    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" (UID: "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:41:09.724752 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.724659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-kube-api-access-5cv8h" (OuterVolumeSpecName: "kube-api-access-5cv8h") pod "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" (UID: "7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e"). InnerVolumeSpecName "kube-api-access-5cv8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:41:09.823279 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.823249    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-token\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:09.823279 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.823274    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-podinfo\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:09.823279 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.823284    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-envoy\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:09.823475 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.823292    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istiod-ca-cert\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:09.823475 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.823300    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-istio-data\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:09.823475 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:09.823308    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cv8h\" (UniqueName: \"kubernetes.io/projected/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e-kube-api-access-5cv8h\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:10.419011 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.418981    2578 generic.go:356] "Generic (PLEG): container finished" podID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerID="18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff" exitCode=0
Jul 28 21:41:10.419213 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.419031    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"
Jul 28 21:41:10.419213 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.419053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" event={"ID":"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e","Type":"ContainerDied","Data":"18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff"}
Jul 28 21:41:10.419213 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.419118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp" event={"ID":"7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e","Type":"ContainerDied","Data":"7896f3a64a6a95489f86620cc551dbf160cfb3d4db466ac8c82b5412a04da5d5"}
Jul 28 21:41:10.419213 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.419142    2578 scope.go:117] "RemoveContainer" containerID="18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff"
Jul 28 21:41:10.428520 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.428504    2578 scope.go:117] "RemoveContainer" containerID="9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83"
Jul 28 21:41:10.437131 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.436874    2578 scope.go:117] "RemoveContainer" containerID="18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff"
Jul 28 21:41:10.437198 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:41:10.437156    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff\": container with ID starting with 18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff not found: ID does not exist" containerID="18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff"
Jul 28 21:41:10.437235 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.437189    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff"} err="failed to get container status \"18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff\": rpc error: code = NotFound desc = could not find container \"18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff\": container with ID starting with 18548269c5a9463497f14c35c687fd98e7a31e00af9639c7bebc16ba86258fff not found: ID does not exist"
Jul 28 21:41:10.437235 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.437207    2578 scope.go:117] "RemoveContainer" containerID="9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83"
Jul 28 21:41:10.437455 ip-10-0-138-164 kubenswrapper[2578]: E0728 21:41:10.437435    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83\": container with ID starting with 9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83 not found: ID does not exist" containerID="9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83"
Jul 28 21:41:10.437512 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.437462    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83"} err="failed to get container status \"9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83\": rpc error: code = NotFound desc = could not find container \"9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83\": container with ID starting with 9ed422675140e7ec3b49e0c1584aef1e10c57908fe18d2da45fba13bf3d59f83 not found: ID does not exist"
Jul 28 21:41:10.442758 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.442737    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"]
Jul 28 21:41:10.446810 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:10.446791    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-35d85b-openshift-default-745f955c6b-d2mvp"]
Jul 28 21:41:11.529931 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:11.529899    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" path="/var/lib/kubelet/pods/7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e/volumes"
Jul 28 21:41:25.217649 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.217615    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"]
Jul 28 21:41:25.218300 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.217863    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd" podUID="f975d585-01de-46e3-be1e-5f3acfb326b2" containerName="payload-processing" containerID="cri-o://8ad76d85c0635831cb7ad65ee27374562c117a2b0531e2f111056753fd2c74c7" gracePeriod=30
Jul 28 21:41:25.228195 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.228167    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"]
Jul 28 21:41:25.229059 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.229004    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt" podUID="c168ec44-10dc-4b12-a22b-3394de38b689" containerName="payload-pre-processing" containerID="cri-o://657efbd2cb7e84dfa79662ef9f63e28f86574884e3ec5325b270521a5963d9aa" gracePeriod=30
Jul 28 21:41:25.498837 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.498804    2578 generic.go:356] "Generic (PLEG): container finished" podID="c168ec44-10dc-4b12-a22b-3394de38b689" containerID="657efbd2cb7e84dfa79662ef9f63e28f86574884e3ec5325b270521a5963d9aa" exitCode=0
Jul 28 21:41:25.499068 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.498900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt" event={"ID":"c168ec44-10dc-4b12-a22b-3394de38b689","Type":"ContainerDied","Data":"657efbd2cb7e84dfa79662ef9f63e28f86574884e3ec5325b270521a5963d9aa"}
Jul 28 21:41:25.500617 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.500461    2578 generic.go:356] "Generic (PLEG): container finished" podID="f975d585-01de-46e3-be1e-5f3acfb326b2" containerID="8ad76d85c0635831cb7ad65ee27374562c117a2b0531e2f111056753fd2c74c7" exitCode=0
Jul 28 21:41:25.500617 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.500495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd" event={"ID":"f975d585-01de-46e3-be1e-5f3acfb326b2","Type":"ContainerDied","Data":"8ad76d85c0635831cb7ad65ee27374562c117a2b0531e2f111056753fd2c74c7"}
Jul 28 21:41:25.500617 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.500534    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd" event={"ID":"f975d585-01de-46e3-be1e-5f3acfb326b2","Type":"ContainerDied","Data":"7d33b826b2adfb4cfa5abb461b9a723b13e272945c0803da2d05cf6e1acd000a"}
Jul 28 21:41:25.500617 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.500557    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d33b826b2adfb4cfa5abb461b9a723b13e272945c0803da2d05cf6e1acd000a"
Jul 28 21:41:25.520256 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.520236    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:41:25.523404 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.523385    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:41:25.575019 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.574992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jt2p\" (UniqueName: \"kubernetes.io/projected/f975d585-01de-46e3-be1e-5f3acfb326b2-kube-api-access-7jt2p\") pod \"f975d585-01de-46e3-be1e-5f3acfb326b2\" (UID: \"f975d585-01de-46e3-be1e-5f3acfb326b2\") "
Jul 28 21:41:25.575203 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.575056    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f975d585-01de-46e3-be1e-5f3acfb326b2-plugins-config-volume\") pod \"f975d585-01de-46e3-be1e-5f3acfb326b2\" (UID: \"f975d585-01de-46e3-be1e-5f3acfb326b2\") "
Jul 28 21:41:25.575203 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.575128    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j92b6\" (UniqueName: \"kubernetes.io/projected/c168ec44-10dc-4b12-a22b-3394de38b689-kube-api-access-j92b6\") pod \"c168ec44-10dc-4b12-a22b-3394de38b689\" (UID: \"c168ec44-10dc-4b12-a22b-3394de38b689\") "
Jul 28 21:41:25.575319 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.575224    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c168ec44-10dc-4b12-a22b-3394de38b689-plugins-config-volume\") pod \"c168ec44-10dc-4b12-a22b-3394de38b689\" (UID: \"c168ec44-10dc-4b12-a22b-3394de38b689\") "
Jul 28 21:41:25.575431 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.575403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f975d585-01de-46e3-be1e-5f3acfb326b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f975d585-01de-46e3-be1e-5f3acfb326b2" (UID: "f975d585-01de-46e3-be1e-5f3acfb326b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:41:25.575567 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.575546    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f975d585-01de-46e3-be1e-5f3acfb326b2-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:25.575901 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.575871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c168ec44-10dc-4b12-a22b-3394de38b689-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c168ec44-10dc-4b12-a22b-3394de38b689" (UID: "c168ec44-10dc-4b12-a22b-3394de38b689"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:41:25.577379 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.577357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f975d585-01de-46e3-be1e-5f3acfb326b2-kube-api-access-7jt2p" (OuterVolumeSpecName: "kube-api-access-7jt2p") pod "f975d585-01de-46e3-be1e-5f3acfb326b2" (UID: "f975d585-01de-46e3-be1e-5f3acfb326b2"). InnerVolumeSpecName "kube-api-access-7jt2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:41:25.577935 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.577915    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c168ec44-10dc-4b12-a22b-3394de38b689-kube-api-access-j92b6" (OuterVolumeSpecName: "kube-api-access-j92b6") pod "c168ec44-10dc-4b12-a22b-3394de38b689" (UID: "c168ec44-10dc-4b12-a22b-3394de38b689"). InnerVolumeSpecName "kube-api-access-j92b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:41:25.676777 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.676735    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j92b6\" (UniqueName: \"kubernetes.io/projected/c168ec44-10dc-4b12-a22b-3394de38b689-kube-api-access-j92b6\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:25.676777 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.676768    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c168ec44-10dc-4b12-a22b-3394de38b689-plugins-config-volume\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:25.676777 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:25.676783    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jt2p\" (UniqueName: \"kubernetes.io/projected/f975d585-01de-46e3-be1e-5f3acfb326b2-kube-api-access-7jt2p\") on node \"ip-10-0-138-164.ec2.internal\" DevicePath \"\""
Jul 28 21:41:26.506465 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:26.506372    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"
Jul 28 21:41:26.506465 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:26.506392    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"
Jul 28 21:41:26.506465 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:26.506416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt" event={"ID":"c168ec44-10dc-4b12-a22b-3394de38b689","Type":"ContainerDied","Data":"eae7b0d74ad16c8d6b90e09662fe0bbd6a3f19b5eb89d266bdb3ed934dfac1e1"}
Jul 28 21:41:26.507000 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:26.506479    2578 scope.go:117] "RemoveContainer" containerID="657efbd2cb7e84dfa79662ef9f63e28f86574884e3ec5325b270521a5963d9aa"
Jul 28 21:41:26.532467 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:26.532444    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"]
Jul 28 21:41:26.535931 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:26.535910    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-7076b1-649fdfdd5d-c5qgd"]
Jul 28 21:41:26.545498 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:26.545479    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"]
Jul 28 21:41:26.549451 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:26.549429    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-7076b1-7757459bc6-d5lbt"]
Jul 28 21:41:27.528720 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:27.528686    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c168ec44-10dc-4b12-a22b-3394de38b689" path="/var/lib/kubelet/pods/c168ec44-10dc-4b12-a22b-3394de38b689/volumes"
Jul 28 21:41:27.529124 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:27.529037    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f975d585-01de-46e3-be1e-5f3acfb326b2" path="/var/lib/kubelet/pods/f975d585-01de-46e3-be1e-5f3acfb326b2/volumes"
Jul 28 21:41:38.815902 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:38.815870    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-76686b4476-w8rl4_d9152958-ebe5-4552-b661-45463b8c6e07/manager/0.log"
Jul 28 21:41:39.168927 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:39.168851    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5966989555-7xnnb_0d6e1a1d-9928-4752-97ca-c3aad3802f6f/copy-manifests/0.log"
Jul 28 21:41:39.175393 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:39.175366    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5966989555-7xnnb_0d6e1a1d-9928-4752-97ca-c3aad3802f6f/manager/0.log"
Jul 28 21:41:40.269518 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:40.269489    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-hmnsl_f416da26-7565-44fc-9ddd-1669731d2be2/postgres/0.log"
Jul 28 21:41:41.759863 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:41.759828    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-fkpjt_8206184c-abd7-4c3f-8235-e27f2215850b/manager/0.log"
Jul 28 21:41:41.884432 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:41.884402    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-4dbnf_66d2cdfb-a082-48d8-83db-571d37580d75/manager/0.log"
Jul 28 21:41:42.130437 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:42.130368    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8vvr6_ca15543a-555e-4a8b-9e8b-d7d9c4e30899/registry-server/0.log"
Jul 28 21:41:42.257437 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:42.257397    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-pnflf_7a35656c-f8be-4560-b2d8-2db4fe7c845d/manager/0.log"
Jul 28 21:41:42.868539 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:42.868510    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p_0684770d-58b4-496c-8a0e-aed1a06baaf8/istio-proxy/0.log"
Jul 28 21:41:43.345432 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:43.345402    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gv6ws_b7de2788-2093-4a2b-bd7c-650e5ab13966/istio-proxy/1.log"
Jul 28 21:41:43.588338 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:43.588309    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-76zbd_73fe5e12-e5b9-4359-8a5b-342ff454649e/payload-processing/0.log"
Jul 28 21:41:44.189125 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:44.189095    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-qxsm2_25c6f3fd-9c63-467e-928a-e15ecb6ec840/main/0.log"
Jul 28 21:41:44.422070 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:44.422020    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgnzbg_e8fe03cc-ec6a-49ca-a7ca-68220d572a1b/main/0.log"
Jul 28 21:41:44.539882 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:44.539850    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-pc4bc_60fa645a-e0e2-484d-b636-6220db27ab89/main/0.log"
Jul 28 21:41:44.655038 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:44.655013    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zgq59_84114904-8707-48e5-85cb-8f576e6e690b/main/0.log"
Jul 28 21:41:53.310136 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:53.310105    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v882q_e665f43b-b228-44a5-a9ff-412e77074d0d/global-pull-secret-syncer/0.log"
Jul 28 21:41:53.393682 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:53.393653    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lx58r_d014f55f-c43a-4ef5-bd1e-e517a72ea8a1/konnectivity-agent/0.log"
Jul 28 21:41:53.477619 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:53.477588    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-164.ec2.internal_18a744e7b35568a34a35b17c19e9750c/haproxy/0.log"
Jul 28 21:41:59.099204 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:59.099121    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-fkpjt_8206184c-abd7-4c3f-8235-e27f2215850b/manager/0.log"
Jul 28 21:41:59.127918 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:59.127897    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-4dbnf_66d2cdfb-a082-48d8-83db-571d37580d75/manager/0.log"
Jul 28 21:41:59.202213 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:59.202194    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8vvr6_ca15543a-555e-4a8b-9e8b-d7d9c4e30899/registry-server/0.log"
Jul 28 21:41:59.362107 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:41:59.362006    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-pnflf_7a35656c-f8be-4560-b2d8-2db4fe7c845d/manager/0.log"
Jul 28 21:42:01.356048 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.355939    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-855b84fc64-nmz64_e5d72cae-f158-41eb-9d7e-60f5633d7466/metrics-server/0.log"
Jul 28 21:42:01.424226 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.424196    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xrrf_cc2a0275-09dc-4108-9557-6cf76e6117b5/node-exporter/0.log"
Jul 28 21:42:01.447294 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.447270    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xrrf_cc2a0275-09dc-4108-9557-6cf76e6117b5/kube-rbac-proxy/0.log"
Jul 28 21:42:01.468814 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.468788    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xrrf_cc2a0275-09dc-4108-9557-6cf76e6117b5/init-textfile/0.log"
Jul 28 21:42:01.655180 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.655098    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-455lz_7b3f5c6a-d31b-415f-9166-af11041f6afc/kube-rbac-proxy-main/0.log"
Jul 28 21:42:01.682107 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.682069    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-455lz_7b3f5c6a-d31b-415f-9166-af11041f6afc/kube-rbac-proxy-self/0.log"
Jul 28 21:42:01.725588 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.725564    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-455lz_7b3f5c6a-d31b-415f-9166-af11041f6afc/openshift-state-metrics/0.log"
Jul 28 21:42:01.776287 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.776250    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a32df3a-e587-48f2-95d5-e35eab0b6272/prometheus/0.log"
Jul 28 21:42:01.800596 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.800561    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a32df3a-e587-48f2-95d5-e35eab0b6272/config-reloader/0.log"
Jul 28 21:42:01.827061 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.827030    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a32df3a-e587-48f2-95d5-e35eab0b6272/thanos-sidecar/0.log"
Jul 28 21:42:01.853353 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.853324    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a32df3a-e587-48f2-95d5-e35eab0b6272/kube-rbac-proxy-web/0.log"
Jul 28 21:42:01.878374 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.878348    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a32df3a-e587-48f2-95d5-e35eab0b6272/kube-rbac-proxy/0.log"
Jul 28 21:42:01.905312 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.905234    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a32df3a-e587-48f2-95d5-e35eab0b6272/kube-rbac-proxy-thanos/0.log"
Jul 28 21:42:01.928320 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:01.928295    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a32df3a-e587-48f2-95d5-e35eab0b6272/init-config-reloader/0.log"
Jul 28 21:42:03.414226 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414149    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"]
Jul 28 21:42:03.414586 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414530    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f90eec3-08ea-44b1-b281-fce597418776" containerName="payload-pre-processing"
Jul 28 21:42:03.414586 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414541    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f90eec3-08ea-44b1-b281-fce597418776" containerName="payload-pre-processing"
Jul 28 21:42:03.414586 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414559    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee98977a-0955-422b-8f19-ffea573d3b90" containerName="payload-processing"
Jul 28 21:42:03.414586 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414565    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee98977a-0955-422b-8f19-ffea573d3b90" containerName="payload-processing"
Jul 28 21:42:03.414586 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414574    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c168ec44-10dc-4b12-a22b-3394de38b689" containerName="payload-pre-processing"
Jul 28 21:42:03.414586 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414579    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c168ec44-10dc-4b12-a22b-3394de38b689" containerName="payload-pre-processing"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414589    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f975d585-01de-46e3-be1e-5f3acfb326b2" containerName="payload-processing"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414595    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f975d585-01de-46e3-be1e-5f3acfb326b2" containerName="payload-processing"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414602    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerName="istio-proxy"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414608    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerName="istio-proxy"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414614    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb3b398-adca-4d22-8777-d1d02f7d0bd2" containerName="payload-pre-processing"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414618    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb3b398-adca-4d22-8777-d1d02f7d0bd2" containerName="payload-pre-processing"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414627    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f0f063a-2325-4bae-a3c7-d320ed50c4e2" containerName="main"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414631    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f0f063a-2325-4bae-a3c7-d320ed50c4e2" containerName="main"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414684    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7478ddd5-4e4d-4b5c-97cc-0cc339e1fa9e" containerName="istio-proxy"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414691    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fb3b398-adca-4d22-8777-d1d02f7d0bd2" containerName="payload-pre-processing"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414699    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c168ec44-10dc-4b12-a22b-3394de38b689" containerName="payload-pre-processing"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414705    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f975d585-01de-46e3-be1e-5f3acfb326b2" containerName="payload-processing"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414711    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f90eec3-08ea-44b1-b281-fce597418776" containerName="payload-pre-processing"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414718    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f0f063a-2325-4bae-a3c7-d320ed50c4e2" containerName="main"
Jul 28 21:42:03.414782 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.414723    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee98977a-0955-422b-8f19-ffea573d3b90" containerName="payload-processing"
Jul 28 21:42:03.418069 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.418054    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.420642 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.420618    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-phbxh\"/\"kube-root-ca.crt\""
Jul 28 21:42:03.420642 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.420641    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-phbxh\"/\"default-dockercfg-xr4cn\""
Jul 28 21:42:03.420894 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.420642    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-phbxh\"/\"openshift-service-ca.crt\""
Jul 28 21:42:03.426602 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.426576    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"]
Jul 28 21:42:03.522766 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.522732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-proc\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.522766 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.522767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-sys\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.523039 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.522789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jl2d\" (UniqueName: \"kubernetes.io/projected/6963100c-5619-418f-ac34-67f15e9098df-kube-api-access-6jl2d\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.523039 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.522809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-lib-modules\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.523039 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.522928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-podres\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.623425 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.623393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-podres\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.623607 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.623487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-proc\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.623607 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.623509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-sys\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.623607 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.623526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jl2d\" (UniqueName: \"kubernetes.io/projected/6963100c-5619-418f-ac34-67f15e9098df-kube-api-access-6jl2d\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.623607 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.623555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-lib-modules\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.623607 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.623578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-podres\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.623867 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.623610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-proc\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.623867 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.623618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-sys\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.623867 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.623714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6963100c-5619-418f-ac34-67f15e9098df-lib-modules\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.631498 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.631466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jl2d\" (UniqueName: \"kubernetes.io/projected/6963100c-5619-418f-ac34-67f15e9098df-kube-api-access-6jl2d\") pod \"perf-node-gather-daemonset-5wxxb\" (UID: \"6963100c-5619-418f-ac34-67f15e9098df\") " pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.728481 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.728444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:03.862986 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:03.862960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"]
Jul 28 21:42:03.865211 ip-10-0-138-164 kubenswrapper[2578]: W0728 21:42:03.865171    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6963100c_5619_418f_ac34_67f15e9098df.slice/crio-83181b3cffe74d5bbd56ead8081f0dd4c7b82baf259718201e8496ba07a6f395 WatchSource:0}: Error finding container 83181b3cffe74d5bbd56ead8081f0dd4c7b82baf259718201e8496ba07a6f395: Status 404 returned error can't find the container with id 83181b3cffe74d5bbd56ead8081f0dd4c7b82baf259718201e8496ba07a6f395
Jul 28 21:42:04.669751 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:04.669724    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-vr4rz_98b94339-faa6-414e-be68-c6ec91e9b4eb/download-server/0.log"
Jul 28 21:42:04.687781 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:04.687751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb" event={"ID":"6963100c-5619-418f-ac34-67f15e9098df","Type":"ContainerStarted","Data":"75d26d26b4707f7e4bfa10c8996ad23122de2dfc0e321fb6ef5deaf71aa575d8"}
Jul 28 21:42:04.687931 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:04.687807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb" event={"ID":"6963100c-5619-418f-ac34-67f15e9098df","Type":"ContainerStarted","Data":"83181b3cffe74d5bbd56ead8081f0dd4c7b82baf259718201e8496ba07a6f395"}
Jul 28 21:42:04.687931 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:04.687895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:04.705411 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:04.705348    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb" podStartSLOduration=1.705334175 podStartE2EDuration="1.705334175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 21:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:42:04.70424665 +0000 UTC m=+3413.559090574" watchObservedRunningTime="2026-07-28 21:42:04.705334175 +0000 UTC m=+3413.560178086"
Jul 28 21:42:06.383843 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:06.383813    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6mjf2_cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961/dns/0.log"
Jul 28 21:42:06.406341 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:06.406312    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6mjf2_cc7e9d2c-fc43-40e4-9412-1a2f1f8f0961/kube-rbac-proxy/0.log"
Jul 28 21:42:06.524500 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:06.524472    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9wb9_87a4da6a-0543-4046-b35d-00579314cafe/dns-node-resolver/0.log"
Jul 28 21:42:07.082143 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:07.082111    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65c598f646-wbnmg_b0bb1835-824f-4d59-abe5-5bd6b74e8b59/registry/0.log"
Jul 28 21:42:07.104329 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:07.104308    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8hrfs_de6d5b34-f052-4613-a09c-4892d9fac92d/node-ca/0.log"
Jul 28 21:42:08.121357 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:08.121323    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p_0684770d-58b4-496c-8a0e-aed1a06baaf8/istio-proxy/0.log"
Jul 28 21:42:08.345157 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:08.345124    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gv6ws_b7de2788-2093-4a2b-bd7c-650e5ab13966/istio-proxy/0.log"
Jul 28 21:42:08.372335 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:08.372243    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gv6ws_b7de2788-2093-4a2b-bd7c-650e5ab13966/istio-proxy/1.log"
Jul 28 21:42:08.434046 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:08.434014    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-76zbd_73fe5e12-e5b9-4359-8a5b-342ff454649e/payload-processing/0.log"
Jul 28 21:42:09.309927 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:09.309896    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xmjzf_acb3e3c8-2af8-4dad-a317-c04e4515711f/serve-healthcheck-canary/0.log"
Jul 28 21:42:09.978478 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:09.978450    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kg2x9_349dc143-219c-4d79-a6a3-24778d73a200/kube-rbac-proxy/0.log"
Jul 28 21:42:10.001687 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:10.001662    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kg2x9_349dc143-219c-4d79-a6a3-24778d73a200/exporter/0.log"
Jul 28 21:42:10.027426 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:10.027398    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kg2x9_349dc143-219c-4d79-a6a3-24778d73a200/extractor/0.log"
Jul 28 21:42:10.702450 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:10.702420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-phbxh/perf-node-gather-daemonset-5wxxb"
Jul 28 21:42:11.920644 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:11.920605    2578 scope.go:117] "RemoveContainer" containerID="8ad76d85c0635831cb7ad65ee27374562c117a2b0531e2f111056753fd2c74c7"
Jul 28 21:42:12.883159 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:12.883129    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-76686b4476-w8rl4_d9152958-ebe5-4552-b661-45463b8c6e07/manager/0.log"
Jul 28 21:42:13.020951 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:13.020921    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5966989555-7xnnb_0d6e1a1d-9928-4752-97ca-c3aad3802f6f/manager/0.log"
Jul 28 21:42:13.045326 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:13.045301    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5966989555-7xnnb_0d6e1a1d-9928-4752-97ca-c3aad3802f6f/copy-manifests/0.log"
Jul 28 21:42:14.047733 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:14.047700    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p_0684770d-58b4-496c-8a0e-aed1a06baaf8/istio-proxy/0.log"
Jul 28 21:42:14.236223 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:14.236190    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gv6ws_b7de2788-2093-4a2b-bd7c-650e5ab13966/istio-proxy/0.log"
Jul 28 21:42:14.264587 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:14.264552    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gv6ws_b7de2788-2093-4a2b-bd7c-650e5ab13966/istio-proxy/1.log"
Jul 28 21:42:14.314524 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:14.314455    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-76zbd_73fe5e12-e5b9-4359-8a5b-342ff454649e/payload-processing/0.log"
Jul 28 21:42:14.540982 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:14.540952    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57d8fcb7f5-vhplp_71f0ed41-6296-4bda-ac5c-d421d3c2febe/manager/0.log"
Jul 28 21:42:14.568558 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:14.568490    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-qhspc_7b17fe68-105a-420a-861c-c21406e07420/openshift-lws-operator/0.log"
Jul 28 21:42:15.703273 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:15.703248    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-m6j8p_0684770d-58b4-496c-8a0e-aed1a06baaf8/istio-proxy/0.log"
Jul 28 21:42:15.892669 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:15.892637    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gv6ws_b7de2788-2093-4a2b-bd7c-650e5ab13966/istio-proxy/0.log"
Jul 28 21:42:15.919888 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:15.919860    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gv6ws_b7de2788-2093-4a2b-bd7c-650e5ab13966/istio-proxy/1.log"
Jul 28 21:42:15.962612 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:15.962513    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-76zbd_73fe5e12-e5b9-4359-8a5b-342ff454649e/payload-processing/0.log"
Jul 28 21:42:19.486627 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:19.486597    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zbxnv_69070dc3-cb63-4714-9542-f866288990d1/migrator/0.log"
Jul 28 21:42:19.505904 ip-10-0-138-164 kubenswrapper[2578]: I0728 21:42:19.505878    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zbxnv_69070dc3-cb63-4714-9542-f866288990d1/graceful-termination/0.log"